thallada
09c87fa93b
Split up batched mod data query to speed up dump
2022-06-05 02:58:11 -04:00
thallada
bafa405223
Add tracing logging to all commands
2022-06-04 18:51:20 -04:00
thallada
026d175da4
Add updated_after option to limit files generated on every run
...
Instead of generating files for all mods every update, only update files for mods that have been updated since the last run.
2022-06-04 00:50:11 -04:00
thallada
bed49d0953
Add a new command to dump files to json files
2022-06-01 00:54:29 -04:00
thallada
7a87f0fd22
Add command to generate mod_cell_counts.json
...
So that I can display cell edit counts in mod lists efficiently.
2022-03-17 23:51:32 -04:00
thallada
a42c22cf4b
Add is_translation to mod with backfill
...
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
5d55e78283
Dump plugins by hash instead of id
...
The same exact plugins are often included in many different files/mods. Grouping by hash in the sql query will show all files and mods a plugin appears in. Previously, the last file that was processed would overwrite the plugin file and only that single file/mod would appear in the file.
Also adds an index on plugins.hash since I'm now querying directly on it.
2022-03-12 13:05:21 -05:00
thallada
c9dcd3b7c5
Include cells with plugin data
2022-03-10 23:27:26 -05:00
thallada
7552573d00
Dump plugins data with file and mod data
2022-03-10 22:32:46 -05:00
thallada
21fffd3c23
Remove dbg logging
2022-02-12 01:25:30 -05:00
thallada
4d9855552b
Add option for full update and default to false
...
When false, stop scraping pages after 50 pages of no new updates
2022-02-11 22:38:44 -05:00
thallada
283cad220b
Move tile_scraper from examples to commands
2022-02-08 00:13:14 -05:00
thallada
b8c5f63aeb
Refactor main.rs into separate files
...
Create new `commands` mod with file for each command line option.
Create a couple more extractor files for moving 7zip and unrar methods into.
2022-02-08 00:02:26 -05:00
thallada
Tyler Hallada