|
|
21fffd3c23
|
Remove dbg logging
|
2022-02-12 01:25:30 -05:00 |
|
|
|
4d9855552b
|
Add option for full update and default to false
When false, stop scraping pages after 50 pages of no new updates
|
2022-02-11 22:38:44 -05:00 |
|
|
|
283cad220b
|
Move tile_scraper from examples to commands
|
2022-02-08 00:13:14 -05:00 |
|
|
|
b8c5f63aeb
|
Refactor main.rs into separate files
Create new `commands` mod with file for each command line option.
Create a couple more extractor files for moving 7zip and unrar methods into.
|
2022-02-08 00:02:26 -05:00 |
|
|
|
51175933d6
|
Fix skyrim worldspace filter in mod cells query
|
2022-02-06 23:31:31 -05:00 |
|
|
|
891dbd791a
|
Limit cells in mod files to Skyrim worldspace cells
|
2022-02-06 23:11:03 -05:00 |
|
|
|
240349cf1a
|
Add option for generating a mod search index for fuse.js
|
2022-01-24 23:20:51 -05:00 |
|
|
|
4875d2d764
|
Use LEFT OUTER JOIN to not skip mods in the dump
|
2022-01-24 22:46:23 -05:00 |
|
|
|
0c3d85d697
|
More efficient batched_get_with_cells
|
2022-01-24 22:33:50 -05:00 |
|
|
|
17facae842
|
Make ancestor cols not nullable, add fk constraints
Also removed the temporary backfill, now that it is complete.
Updates models so cols are not optional.
|
2022-01-24 21:41:45 -05:00 |
|
|
|
e5936683fa
|
Add ancestor id columns to plugins and plugin_cells
With temp backfill queries. Second stage will to make the columns nullable and add foreign keys.
I'm hoping that this will improve query performance.
|
2022-01-24 21:14:59 -05:00 |
|
|
|
4a8e3cad6c
|
Add option for generating mod files
|
2022-01-24 01:24:15 -05:00 |
|
|
|
e7fe7abfb7
|
Set columns to not null, remove backfill code
|
2022-01-18 21:50:09 -05:00 |
|
|
|
337b673e02
|
Tolerate null picture_url in response
|
2022-01-18 21:03:26 -05:00 |
|
|
|
d48e2fdb5f
|
Add option to temporarily backfill mod data
|
2022-01-18 20:54:50 -05:00 |
|
|
|
f22587d4a0
|
Handle missing category_id in scraper
|
2022-01-18 17:03:22 -05:00 |
|
|
|
e7b5c750cd
|
Temporarily update all mods to backfill fields
|
2022-01-17 23:55:22 -05:00 |
|
|
|
b97689b7fd
|
Scrape additional fields to mod table
|
2022-01-17 23:37:58 -05:00 |
|
|
|
f23cf526e5
|
Fix duplicate mods in cell data
|
2022-01-17 20:37:06 -05:00 |
|
|
|
a78fa4772b
|
Add cli option to save cell data json to directory
|
2022-01-17 19:25:07 -05:00 |
|
Tyler Hallada
|
e779e94eff
|
Add -e option to output cell edits from database
|
2022-01-15 00:19:05 -05:00 |
|
|
|
804fac4eea
|
Start processing mod updates
|
2021-09-05 16:02:16 -04:00 |
|
Tyler Hallada
|
ba6ca0c928
|
Fix clippy lints, add 7z fallback for failed unrar
|
2021-08-14 11:43:09 -06:00 |
|
Tyler Hallada
|
be9c78217d
|
Skip files with unknown file type
|
2021-08-12 10:25:28 -06:00 |
|
Tyler Hallada
|
85f1d3d05a
|
Recover failed 7z uncompress too
|
2021-08-11 10:10:02 -06:00 |
|
Tyler Hallada
|
1169ef4536
|
Print human readable file size to log
|
2021-08-09 22:38:17 -06:00 |
|
Tyler Hallada
|
db2e73aa8e
|
Skip unrar extract errors, update compress-tools
|
2021-08-09 11:10:57 -06:00 |
|
Tyler Hallada
|
6136ac18fa
|
Make mod category nullable
|
2021-08-08 19:14:23 -06:00 |
|
|
|
b2d17f6217
|
Refactor main.rs a bit, some fixes to 7zip fallback
Also adds reqwest timeouts
|
2021-08-07 00:53:56 -04:00 |
|
|
|
3f48f97080
|
Fix typo
|
2021-07-29 23:45:18 -04:00 |
|
|
|
3ec7467571
|
Add more status columns to fix continuation of unprocessed files
|
2021-07-29 22:55:34 -04:00 |
|
|
|
fc5a9d39c2
|
Add timestamp to mods for some resumability
|
2021-07-29 00:50:42 -04:00 |
|
|
|
a8424e830e
|
Adds download_at to files, skips files that are impossible to download
|
2021-07-28 23:54:39 -04:00 |
|
|
|
8a356ac7f5
|
Retry connect failures, write plugins to disk instead of zip archive
Writing to the zip was starting to take forever. It makes more sense to just use my big HD and then zip after I'm done downloading every file.
|
2021-07-26 19:31:25 -04:00 |
|
|
|
f62324d36c
|
Reset main.rs back w/ fix to listing RAR files
|
2021-07-26 17:02:14 -04:00 |
|
|
|
5d62fb7d61
|
Fix plugin listing for rar files, temporary backfil shim
`main` is just backfilling the eixsting bad rar files in the db. I will return script to normal once it's run.
|
2021-07-26 01:16:43 -04:00 |
|
|
|
87ca90be06
|
Fix plugin insert SQL binding
|
2021-07-25 18:33:45 -04:00 |
|
|
|
523f3eeb3f
|
Add retry logic to metadata fetch, don't crash on 500
|
2021-07-25 18:03:54 -04:00 |
|
|
|
39ae7703b0
|
Upgrade skyrim-cell-dump and handle Cow<str>s
|
2021-07-23 23:40:05 -04:00 |
|
|
|
19126f4981
|
Add retry logic to API calls
Since apparently these can 500 occasionally.
|
2021-07-23 22:29:28 -04:00 |
|
|
|
81f12da99c
|
Upgrade unrar to master branch to fix Utf8Error
|
2021-07-23 19:34:03 -04:00 |
|
|
|
e69165fe5a
|
Set page size to 20 since nexus mods doesn't always obey 80
|
2021-07-22 21:55:31 -04:00 |
|
|
|
77562a7a05
|
Add CLI args with page, skip unreadable archives
|
2021-07-22 21:50:28 -04:00 |
|
|
|
e534447310
|
Simplify bulk_get_present_nexus_mod_ids query
|
2021-07-22 00:13:20 -04:00 |
|
|
|
f337917c99
|
Batch insert mods, add lifetimes to batch inserts
Using `&str`s prevents unneeded cloning everywhere, but unfortunately it doesn't work with the sqlx macros.
|
2021-07-21 23:20:51 -04:00 |
|
|
|
04773954f9
|
Remove backfill, add bulk get game_mod method
|
2021-07-21 22:25:45 -04:00 |
|
|
|
4b333b3b99
|
Add world tables and columns, temporary backfill
Too lazy to make a new script, main() has a backfill using plugins.zip. Once I run it I will remove it.
|
2021-07-21 21:35:11 -04:00 |
|
|
|
0e0fdfd59d
|
Migration and models for worlds/plugin_worlds
|
2021-07-19 23:47:00 -04:00 |
|
|
|
1fd979b8e0
|
Skip archived files which cannot be downloaded
|
2021-07-18 22:11:03 -04:00 |
|
|
|
f6a7490214
|
Skip empty or invalid file archives
|
2021-07-18 16:06:49 -04:00 |
|
