thallada/modmapper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Downloads every Skyrim mod plugin from nexusmods.com and saves CELL edits of each to a database https://modmapper.com
108 Commits 1 Branch 0 Tags 490 KiB
Rust 98.1%
Shell 1.9%
14be03cd3d
Go to file
Tyler Hallada 14be03cd3d Make separate mod search indices per game instead 
Saves significant amount of space in the json if I don't need to list a game name for every mod.
2022-09-03 00:49:57 -04:00
migrations Add is_translation to mod with backfill 2022-03-15 23:59:56 -04:00
scripts Make separate mod search indices per game instead 2022-09-03 00:49:57 -04:00
src Make separate mod search indices per game instead 2022-09-03 00:49:57 -04:00
.gitignore Add tracing logging to all commands 2022-06-04 18:51:20 -04:00
Cargo.lock Serialize plugin hashes to strings 2022-08-08 23:37:27 -04:00
Cargo.toml Add cli option to save cell data json to directory 2022-01-17 19:25:07 -05:00
drop_all.sql Rename drop_all.sql file 2021-07-29 01:01:27 -04:00
README.md Add support for scraping classic Skyrim mods 2022-09-02 00:43:53 -04:00

README.md

modmapper

Downloads mods from nexus, parses the plugins inside, and saves data to a postgres database.

Development Install

  1. Install and run postgres.
  2. Create postgres user and database (and add uuid extension while you're there ):
createuser modmapper
createdb modmapper
sudo -u postgres -i psql
postgres=# ALTER DATABASE modmapper OWNER TO modmapper;
\password modmapper

# Or, on Windows in PowerShell:

& 'C:\Program Files\PostgreSQL\13\bin\createuser.exe' -U postgres modmapper
& 'C:\Program Files\PostgreSQL\13\bin\createdb.exe' -U postgres modmapper
& 'C:\Program Files\PostgreSQL\13\bin\psql.exe' -U postgres
postgres=# ALTER DATABASE modmapper OWNER TO modmapper;
\password modmapper
  1. Save password somewhere safe and then and add a .env file to the project directory with the contents:
DATABASE_URL=postgresql://modmapper:<password>@localhost/modmapper
RUST_LOG=mod_mapper=debug
  1. Install sqlx_cli with cargo install sqlx-cli --no-default-features --features postgres
  2. Run sqlx migrate --source migrations run which will run all the database migrations.
  3. Get your personal Nexus API token from your profile settings and add it to the .env file:
NEXUS_API_KEY=...
  1. Build the release binary by running cargo build --release.
  2. See ./target/release/modmapper -h for further commands or run ./scripts/update.sh to start populating the database with scraped mods and dumping the data to JSON files.

Sync and Backup Setup

scripts/sync.sh and scripts/backup.sh both utilize rclone to transfer files that are generated on the machine running modmapper to separate servers for file storage.

For these scripts to run successfully you will need to install rclone and setup a remote for sync.sh (the "static server") and a remote for backup.sh (the "backup server"). Remotes can be created with the rclone config command. Then, make sure these variables are defined in the .env file corresponding to the remote names and buckets (or folders) within that remote you created:

  • STATIC_SERVER_REMOTE
  • STATIC_SERVER_CELLS_BUCKET
  • STATIC_SERVER_MODS_BUCKET
  • STATIC_SERVER_PLUGINS_BUCKET
  • STATIC_SERVER_FILES_BUCKET
  • BACKUP_SERVER_REMOTE
  • BACKUP_SERVER_BUCKET