56864c7f29
Fix mode page scraping
...
Was returning 403s for the past month.
2023-06-30 12:30:12 -04:00
03cf20029e
Revert "Login before scraping mods to include adult mods"
...
This reverts commit bb02771b87
2022-10-02 02:37:15 -04:00
bb02771b87
Login before scraping mods to include adult mods
2022-10-02 00:34:38 -04:00
cf4e83c054
Fix has_next_page scraping logic
...
The next button doesn't appear on the last four pages. Instead, check if the selected page is the last page.
2022-09-26 00:15:25 -04:00
203af0adc9
Scope bulk_get_last_updated_by_nexus_mod_ids by game_id
...
Was accidentally skipping a ton of skyrim mods because of this.
2022-09-26 00:03:39 -04:00
f2a4205b89
Handle missing author names
...
For some reason, some older Skyrim mods have an author link but no author text. In that case, default the name to "Unknown".
2022-09-17 12:25:59 -04:00
7d229ccd1a
Add support for scraping classic Skyrim mods
...
Not adding it to the update.sh script quite yet since I need to run the very long process of downloading every classic skyrim mod that currently exists on the nexus first.
2022-09-02 00:43:53 -04:00
a42c22cf4b
Add is_translation to mod with backfill
...
Now, the update command will scrape all non-translation mods and translation mods separately so the is_translation value can be set correctly in the future.
2022-03-15 23:59:56 -04:00
f22587d4a0
Handle missing category_id in scraper
2022-01-18 17:03:22 -05:00
b97689b7fd
Scrape additional fields to mod table
2022-01-17 23:37:58 -05:00
804fac4eea
Start processing mod updates
2021-09-05 16:02:16 -04:00
Tyler Hallada
ba6ca0c928
Fix clippy lints, add 7z fallback for failed unrar
2021-08-14 11:43:09 -06:00
Tyler Hallada
6136ac18fa
Make mod category nullable
2021-08-08 19:14:23 -06:00
e69165fe5a
Set page size to 20 since nexus mods doesn't always obey 80
2021-07-22 21:55:31 -04:00
77562a7a05
Add CLI args with page, skip unreadable archives
2021-07-22 21:50:28 -04:00
792e78391c
Replace dbg macros with tracing events
2021-07-11 19:45:26 -04:00
22757bc475
Modularize scraping and api requests
2021-07-09 00:37:08 -04:00
