04c0501514
Revert change to get_cell_data query
...
I didn't meant to commit this, it was an experiment that ended up being way too slow.
2022-03-23 17:24:41 -04:00
4e5284a57f
Add mod_cell_counts dump to update.sh
2022-03-18 00:30:27 -04:00
65f841eef3
Go back to INNER JOIN on mod_cell_counts instead
...
This saves a lot of space and there's no real need to differentiate between 0 and does not exist.
2022-03-18 00:28:23 -04:00
02c03bd1d9
Use LEFT OUTER JOIN to include mods with 0 cell edits
2022-03-18 00:16:33 -04:00
7a87f0fd22
Add command to generate mod_cell_counts.json
...
So that I can display cell edit counts in mod lists efficiently.
2022-03-17 23:51:32 -04:00
a42c22cf4b
Add is_translation to mod with backfill
...
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
Tyler Hallada
5d55e78283
Dump plugins by hash instead of id
...
The same exact plugins are often included in many different files/mods. Grouping by hash in the sql query will show all files and mods a plugin appears in. Previously, the last file that was processed would overwrite the plugin file and only that single file/mod would appear in the file.
Also adds an index on plugins.hash since I'm now querying directly on it.
2022-03-12 13:05:21 -05:00
c9dcd3b7c5
Include cells with plugin data
2022-03-10 23:27:26 -05:00
7552573d00
Dump plugins data with file and mod data
2022-03-10 22:32:46 -05:00
17f766da29
Fix rclone command in backup.sh
2022-02-13 14:34:09 -05:00
bb2059e697
Fix find command options in backup.sh
2022-02-12 23:26:45 -05:00
21fffd3c23
Remove dbg logging
2022-02-12 01:25:30 -05:00
eb92f7191d
Add scripts for updating and backing up data
2022-02-12 00:44:32 -05:00
4d9855552b
Add option for full update and default to false
...
When false, stop scraping pages after 50 pages of no new updates
2022-02-11 22:38:44 -05:00
283cad220b
Move tile_scraper from examples to commands
2022-02-08 00:13:14 -05:00
b8c5f63aeb
Refactor main.rs into separate files
...
Create new `commands` mod with file for each command line option.
Create a couple more extractor files for moving 7zip and unrar methods into.
2022-02-08 00:02:26 -05:00
51175933d6
Fix skyrim worldspace filter in mod cells query
2022-02-06 23:31:31 -05:00
82996f237f
Add missing migration for cell lookup index
2022-02-06 23:30:20 -05:00
891dbd791a
Limit cells in mod files to Skyrim worldspace cells
2022-02-06 23:11:03 -05:00
240349cf1a
Add option for generating a mod search index for fuse.js
2022-01-24 23:20:51 -05:00
4875d2d764
Use LEFT OUTER JOIN to not skip mods in the dump
2022-01-24 22:46:23 -05:00
0c3d85d697
More efficient batched_get_with_cells
2022-01-24 22:33:50 -05:00
17facae842
Make ancestor cols not nullable, add fk constraints
...
Also removed the temporary backfill, now that it is complete.
Updates models so cols are not optional.
2022-01-24 21:41:45 -05:00
10c299dc45
Add indices for new ancestor id columns
2022-01-24 21:21:12 -05:00
e5936683fa
Add ancestor id columns to plugins and plugin_cells
...
With temp backfill queries. Second stage will to make the columns nullable and add foreign keys.
I'm hoping that this will improve query performance.
2022-01-24 21:14:59 -05:00
4a8e3cad6c
Add option for generating mod files
2022-01-24 01:24:15 -05:00
e7fe7abfb7
Set columns to not null, remove backfill code
2022-01-18 21:50:09 -05:00
337b673e02
Tolerate null picture_url in response
2022-01-18 21:03:26 -05:00
d48e2fdb5f
Add option to temporarily backfill mod data
2022-01-18 20:54:50 -05:00
f22587d4a0
Handle missing category_id in scraper
2022-01-18 17:03:22 -05:00
e7b5c750cd
Temporarily update all mods to backfill fields
2022-01-17 23:55:22 -05:00
b97689b7fd
Scrape additional fields to mod table
2022-01-17 23:37:58 -05:00
f23cf526e5
Fix duplicate mods in cell data
2022-01-17 20:37:06 -05:00
a78fa4772b
Add cli option to save cell data json to directory
2022-01-17 19:25:07 -05:00
Tyler Hallada
e779e94eff
Add -e option to output cell edits from database
2022-01-15 00:19:05 -05:00
Tyler Hallada
dad58f6154
Add UESP map tile scraper as an example
2022-01-15 00:18:30 -05:00
804fac4eea
Start processing mod updates
2021-09-05 16:02:16 -04:00
0989477b40
Upgrade skyrim-cell-dump to 0.3.1
2021-08-27 00:48:03 -04:00
Tyler Hallada
ba6ca0c928
Fix clippy lints, add 7z fallback for failed unrar
2021-08-14 11:43:09 -06:00
Tyler Hallada
be9c78217d
Skip files with unknown file type
2021-08-12 10:25:28 -06:00
Tyler Hallada
85f1d3d05a
Recover failed 7z uncompress too
2021-08-11 10:10:02 -06:00
Tyler Hallada
1169ef4536
Print human readable file size to log
2021-08-09 22:38:17 -06:00
Tyler Hallada
db2e73aa8e
Skip unrar extract errors, update compress-tools
2021-08-09 11:10:57 -06:00
Tyler Hallada
6136ac18fa
Make mod category nullable
2021-08-08 19:14:23 -06:00
b2d17f6217
Refactor main.rs a bit, some fixes to 7zip fallback
...
Also adds reqwest timeouts
2021-08-07 00:53:56 -04:00
3f48f97080
Fix typo
2021-07-29 23:45:18 -04:00
3ec7467571
Add more status columns to fix continuation of unprocessed files
2021-07-29 22:55:34 -04:00
e1e7dd2e5d
Rename drop_all.sql file
2021-07-29 01:01:27 -04:00
fc5a9d39c2
Add timestamp to mods for some resumability
2021-07-29 00:50:42 -04:00
a8424e830e
Adds download_at to files, skips files that are impossible to download
2021-07-28 23:54:39 -04:00
