Downloads every Skyrim mod plugin from nexusmods.com and saves CELL edits of each to a database https://modmapper.com

Go to file

Tyler Hallada e85edff30a Fix bad interior cell data in db I had a bad unique index on the `cells` table. This commit does not fix it but has two backfill scripts that will allow me to fixup the data before creating the new unique index. `is_base_game` identifies cells that are from the `Skyrim.esm` shipped with the game (and not from some mod that happens to also have a `Skyrim.esm` with a 0,0 cell with a different form id, see: https://github.com/thallada/modmapper-web/issues/8). `dump_cell_data` now only dumps cells from `is_base_game` cells. `deduplicate_interior_cells` removes duplicate cell and plugin_cell rows caused by the buggy index. I will need to upgrade the database and create a new index in a later commit.		2023-11-08 01:10:49 -05:00
data	Fix bad interior cell data in db	2023-11-08 01:10:49 -05:00
migrations	Fix bad interior cell data in db	2023-11-08 01:10:49 -05:00
scripts	Enable updating skyrim mods	2022-09-24 19:58:26 -04:00
src	Fix bad interior cell data in db	2023-11-08 01:10:49 -05:00
.gitignore	Add tracing logging to all commands	2022-06-04 18:51:20 -04:00
Cargo.lock	Fix bad interior cell data in db	2023-11-08 01:10:49 -05:00
Cargo.toml	Update deps, rustfmt, clippy fixes	2023-04-20 00:47:14 -04:00
drop_all.sql	Rename drop_all.sql file	2021-07-29 01:01:27 -04:00
README.md	Fix bad interior cell data in db	2023-11-08 01:10:49 -05:00

README.md

modmapper

Downloads mods from nexus, parses the plugins inside, and saves data to a postgres database.

Development Install

Install and run postgres.
Create postgres user and database (and add uuid extension while you're there ):

createuser modmapper
createdb modmapper
sudo -u postgres -i psql
postgres=# ALTER DATABASE modmapper OWNER TO modmapper;
\password modmapper

# Or, on Windows in PowerShell:

& 'C:\Program Files\PostgreSQL\13\bin\createuser.exe' -U postgres modmapper
& 'C:\Program Files\PostgreSQL\13\bin\createdb.exe' -U postgres modmapper
& 'C:\Program Files\PostgreSQL\13\bin\psql.exe' -U postgres
postgres=# ALTER DATABASE modmapper OWNER TO modmapper;
postgres=# ALTER USER modmapper CREATEDB;
\password modmapper

Save password somewhere safe and then and add a .env file to the project directory with the contents:

DATABASE_URL=postgresql://modmapper:<password>@localhost/modmapper
RUST_LOG=mod_mapper=debug

Install sqlx_cli with cargo install sqlx-cli --no-default-features --features postgres
Run sqlx migrate --source migrations run which will run all the database migrations.
Get your personal Nexus API token from your profile settings and add it to the .env file:

NEXUS_API_KEY=...

Build the release binary by running cargo build --release.
Run ./target/release/modmapper --backfill-is-game-cell to pre-populate the database with worlds and cells from the base game's Skyrim.esm. (This is so that the base game cells can later be differentiated from cells in plugins that also happen to be named Skyrim.esm and have cells that reference a world with the same form ID as Tamriel.)
See ./target/release/modmapper -h for further commands or run ./scripts/update.sh to start populating the database with scraped mods and dumping the data to JSON files.

Sync and Backup Setup

scripts/sync.sh and scripts/backup.sh both utilize rclone to transfer files that are generated on the machine running modmapper to separate servers for file storage.

For these scripts to run successfully you will need to install rclone and setup a remote for sync.sh (the "static server") and a remote for backup.sh (the "backup server"). Remotes can be created with the rclone config command. Then, make sure these variables are defined in the .env file corresponding to the remote names and buckets (or folders) within that remote you created:

STATIC_SERVER_REMOTE
STATIC_SERVER_CELLS_BUCKET
STATIC_SERVER_MODS_BUCKET
STATIC_SERVER_PLUGINS_BUCKET
STATIC_SERVER_FILES_BUCKET
BACKUP_SERVER_REMOTE
BACKUP_SERVER_BUCKET