thallada
65f841eef3
Go back to INNER JOIN on mod_cell_counts instead
...
This saves a lot of space and there's no real need to differentiate between 0 and does not exist.
2022-03-18 00:28:23 -04:00
thallada
02c03bd1d9
Use LEFT OUTER JOIN to include mods with 0 cell edits
2022-03-18 00:16:33 -04:00
thallada
7a87f0fd22
Add command to generate mod_cell_counts.json
...
So that I can display cell edit counts in mod lists efficiently.
2022-03-17 23:51:32 -04:00
thallada
a42c22cf4b
Add is_translation to mod with backfill
...
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
5d55e78283
Dump plugins by hash instead of id
...
The same exact plugins are often included in many different files/mods. Grouping by hash in the sql query will show all files and mods a plugin appears in. Previously, the last file that was processed would overwrite the plugin file and only that single file/mod would appear in the file.
Also adds an index on plugins.hash since I'm now querying directly on it.
2022-03-12 13:05:21 -05:00
thallada
c9dcd3b7c5
Include cells with plugin data
2022-03-10 23:27:26 -05:00
thallada
7552573d00
Dump plugins data with file and mod data
2022-03-10 22:32:46 -05:00
thallada
17f766da29
Fix rclone command in backup.sh
2022-02-13 14:34:09 -05:00
thallada
bb2059e697
Fix find command options in backup.sh
2022-02-12 23:26:45 -05:00
thallada
21fffd3c23
Remove dbg logging
2022-02-12 01:25:30 -05:00
thallada
eb92f7191d
Add scripts for updating and backing up data
2022-02-12 00:44:32 -05:00
thallada
4d9855552b
Add option for full update and default to false
...
When false, stop scraping pages after 50 pages of no new updates
2022-02-11 22:38:44 -05:00
thallada
283cad220b
Move tile_scraper from examples to commands
2022-02-08 00:13:14 -05:00
thallada
b8c5f63aeb
Refactor main.rs into separate files
...
Create new `commands` mod with file for each command line option.
Create a couple more extractor files for moving 7zip and unrar methods into.
2022-02-08 00:02:26 -05:00
thallada
51175933d6
Fix skyrim worldspace filter in mod cells query
2022-02-06 23:31:31 -05:00
thallada
82996f237f
Add missing migration for cell lookup index
2022-02-06 23:30:20 -05:00
thallada
891dbd791a
Limit cells in mod files to Skyrim worldspace cells
2022-02-06 23:11:03 -05:00
thallada
240349cf1a
Add option for generating a mod search index for fuse.js
2022-01-24 23:20:51 -05:00
thallada
4875d2d764
Use LEFT OUTER JOIN to not skip mods in the dump
2022-01-24 22:46:23 -05:00
thallada
0c3d85d697
More efficient batched_get_with_cells
2022-01-24 22:33:50 -05:00
thallada
17facae842
Make ancestor cols not nullable, add fk constraints
...
Also removed the temporary backfill, now that it is complete.
Updates models so cols are not optional.
2022-01-24 21:41:45 -05:00
thallada
10c299dc45
Add indices for new ancestor id columns
2022-01-24 21:21:12 -05:00
thallada
e5936683fa
Add ancestor id columns to plugins and plugin_cells
...
With temp backfill queries. Second stage will to make the columns nullable and add foreign keys.
I'm hoping that this will improve query performance.
2022-01-24 21:14:59 -05:00
thallada
4a8e3cad6c
Add option for generating mod files
2022-01-24 01:24:15 -05:00
thallada
e7fe7abfb7
Set columns to not null, remove backfill code
2022-01-18 21:50:09 -05:00
thallada
337b673e02
Tolerate null picture_url in response
2022-01-18 21:03:26 -05:00
thallada
d48e2fdb5f
Add option to temporarily backfill mod data
2022-01-18 20:54:50 -05:00
thallada
f22587d4a0
Handle missing category_id in scraper
2022-01-18 17:03:22 -05:00
thallada
e7b5c750cd
Temporarily update all mods to backfill fields
2022-01-17 23:55:22 -05:00
thallada
b97689b7fd
Scrape additional fields to mod table
2022-01-17 23:37:58 -05:00
thallada
f23cf526e5
Fix duplicate mods in cell data
2022-01-17 20:37:06 -05:00
thallada
a78fa4772b
Add cli option to save cell data json to directory
2022-01-17 19:25:07 -05:00
e779e94eff
Add -e option to output cell edits from database
2022-01-15 00:19:05 -05:00
dad58f6154
Add UESP map tile scraper as an example
2022-01-15 00:18:30 -05:00
thallada
804fac4eea
Start processing mod updates
2021-09-05 16:02:16 -04:00
thallada
0989477b40
Upgrade skyrim-cell-dump to 0.3.1
2021-08-27 00:48:03 -04:00
ba6ca0c928
Fix clippy lints, add 7z fallback for failed unrar
2021-08-14 11:43:09 -06:00
be9c78217d
Skip files with unknown file type
2021-08-12 10:25:28 -06:00
85f1d3d05a
Recover failed 7z uncompress too
2021-08-11 10:10:02 -06:00
1169ef4536
Print human readable file size to log
2021-08-09 22:38:17 -06:00
db2e73aa8e
Skip unrar extract errors, update compress-tools
2021-08-09 11:10:57 -06:00
6136ac18fa
Make mod category nullable
2021-08-08 19:14:23 -06:00
thallada
b2d17f6217
Refactor main.rs a bit, some fixes to 7zip fallback
...
Also adds reqwest timeouts
2021-08-07 00:53:56 -04:00
thallada
3f48f97080
Fix typo
2021-07-29 23:45:18 -04:00
thallada
3ec7467571
Add more status columns to fix continuation of unprocessed files
2021-07-29 22:55:34 -04:00
thallada
e1e7dd2e5d
Rename drop_all.sql file
2021-07-29 01:01:27 -04:00
thallada
fc5a9d39c2
Add timestamp to mods for some resumability
2021-07-29 00:50:42 -04:00
thallada
a8424e830e
Adds download_at to files, skips files that are impossible to download
2021-07-28 23:54:39 -04:00
thallada
8a356ac7f5
Retry connect failures, write plugins to disk instead of zip archive
...
Writing to the zip was starting to take forever. It makes more sense to just use my big HD and then zip after I'm done downloading every file.
2021-07-26 19:31:25 -04:00
thallada
f62324d36c
Reset main.rs back w/ fix to listing RAR files
2021-07-26 17:02:14 -04:00
thallada
Tyler Hallada