thallada.github.io/_posts/2022-10-05-modmapper-putting-every-skyrim-mod-on-a-map-with-rust.md

title	layout	image
Modmapper: Putting every Skyrim mod on a map with Rust	post	/img/blog/modmapper.jpg

Modmapper is a website that I made that puts every mod for the game Elder Scrolls V: Skyrim uploaded to Nexus Mods on an interactive map.


You can view the map at https://modmapper.com.

Released in 2011, Skyrim is over a decade old now. But, its vast modding community has kept it alive and relevant to this day. Skyrim is still in the top 50 games being played on Steam in 2022 and I think it's no coincidence that it's also one of the most modded games ever.

The enormous and enduring modding community around the Elder Scrolls games is why I have a special fondness for the series. I was 13 when I first got interested in programming through making mods for Elder Scrolls IV: Oblivion. I quickly realized I got way more satisfaction out of modding the game than actually playing it. I was addicted to being able to create whatever my mind imagined in my favorite game.

I was working on mod for Skyrim earlier in the year¹ and was looking for the best places to put new buildings in the game world. I really wanted areas of the game world off the beaten (heavily-modded) path. After over a decade of modifications, there could be conflicts with hundreds of mods in any area I chose which could cause issues like multiple buildings overlapping or terrain changes causing floating rocks and trees.

Example of a conflict between two mods that both chose the same spot to put a lamp post and sign post so they are clipping. Screenshot by AndreySG.

Conflict between a building and rock. Screenshot by LewdManoSaurus.

Conflict between a building and a tree. Screenshot by Janquel.

Conflict between two woodcutting mills. Screenshot by Janquel.

Mod authors usually use a tool like TES5Edit to analyze a group of mod plugins to find conflicts and create patches to resolve them on a case-by-case basis. But, I was unsatisfied with that. I wanted to be assured that there would be no conflicts, or at least know the set of all possible mods out there that could conflict so I could manually patch those few mods. There was no good solution for finding conflicts across all mods though. Mod authors would need to download every Skyrim mod ever and no one has time to download all 85,000+ Skyrim mods, and no one has the computer memory to load all of those in TES5Edit at the same time.

Through that frustration, Modmapper was born with the mission to create a database of all Skyrim mod exterior cell edits. With that database I can power the website which visualizes how popular cells are in aggregate as well as allow the user to drill down to individual cells, mods, or plugins to find potential conflicts without ever having to download files themselves.

When I released the website about 7 months ago it made a big splash in the Skyrim modding community. No one had ever visualized mods on a map like this before, and it gave everyone a new perspective on the vast library of Skyrim mods. It was even featured on the front page of PC Gamer's website. Thirteen-year-old me, who regularly read the monthly PC Gamer magazine, would have been astounded.

The comments posted to the initial mod I posted on Nexus Mods² for the project were very amusing. It seemed to be blowing their minds:

"Quite possibly this could be the best mod for Skyrim. This hands-down makes everyone's life easier to be able to see which of their mods might be conflicting." -- Nexus Mods comment by lorddonk

"The 8th wonder of Skyrim. That's a Titan's work requiring a monk's perserverance. Finally, a place to go check (in)compatibilities !!! Voted. Endorsed." -- Nexus Mods comment by jfjb2005

"They shall sing songs of your greatness! Wow, just wow." -- Nexus Mods comment by LumenMystic

"Holy Batman Tits! Be honest..... You're a Govt Agent and made this mod during your "Terrorist Watch Shift" using a CIA super computer.." -- Nexus Mods comment by toddrizzle

"What drugs are you on and can I have some?" -- Nexus Mods comment by thappysnek

"This is madness! Author are some kind of overhuman?! GREAT work!"-- Nexus Mods comment by TeodorWild

"You are an absolute legend. Bards will sing tales of your exploits" -- Nexus Mods comment by burntwater

"I wanted to say something, but I'll just kneel before thee and worship. This would have taken me a lifetime. Amazing." -- Nexus Mods comment by BlueGunk

"Finally found the real dragonborn" -- Nexus Mods comment by yag1z

"he is the messiah!" -- Nexus Mods comment by Cursedobjects

"A god amongst men." -- Nexus Mods comment by TheMotherRobbit

Apparently knowing how to program is now a god-like ability! This is the type of feedback most programmers aspire to get from their users. I knew the tool was neat and fun to build, but I didn't realize it was that sorely needed by the community.

Today, Modmapper has a sustained user-base of around 7.5k unique visitors a month³ and I still see it mentioned in reddit threads or discord servers whenever someone is asking about the places a mod edits or what mods might be conflicting in a particular cell.

The rest of this blog post will delve into how I built the website and how I gathered all of the data necessary to display the visualization.

Downloading ALL THE MODS!

In order for the project to work I needed to collect all the Skyrim mod plugin files.

While there are a number of places people upload Skyrim mods, Nexus Mods is conveniently the most popular and has the vast majority of mods. So, I would only need to deal with this one source. Luckily, they have a nice API handy.

modmapper is the project I created to do this. It is a Rust binary that:

• Uses reqwest to make requests to Nexus Mods for pages of last updated mods.
• Uses scraper to scrape the HTML for individual mod metadata (since the Nexus API doesn't provide an endpoint to list mods).
• Makes requests to the Nexus Mods API to get file and download information for each mod, using serde to parse the JSON responses.
• Requests the content preview data for each file and walks through the list of files in the archive looking for a Skyrim plugin file (.esp, .esm, or .esl).
• If it finds a plugin, it decides to download the mod. It hits the download API to get a download link and downloads the mod file archive.
• Then it extracts the archive using one of: compress_tools, unrar, or 7zip via std::process::Commmand (depending on what type of archive it is).
• With the ESP files (Elder Scrolls Plugin files) extracted, I then use my skyrim-cell-dump library (more on that later!) to extract all of the cell edits into structured data.
• Uses seahash to create a fast unique hash for plugin files.
• It then saves all of this data to a postgres database using the sqlx crate.
• Uses extensive logging with the tracing crate so I can monitor the output and have a history of a run to debug later if I discover an issue.

It is designed to be run as a nightly cron job which downloads mods that have updated on Nexus Mods since the last run.

To keep costs for this project low, I decided to make the website entirely static. So, instead of creating an API server that would have to be constantly running to serve requests from the website by making queries directly to the database, I would dump all of the data that the website needed from the database to JSON files, then upload those files to Amazon S3 and serve them through the Cloudflare CDN which has servers all over the world.

So, for example, every mod in the database has a JSON file uploaded to https://mods.modmapper.com/skyrimspecialedition/<nexus_mod_id>.json and the website frontend will fetch that file when a user clicks a link to that mod in the UI.

The cost for S3 is pretty reasonable to me ($~3.5/month), and Cloudflare has a generous free tier that allows me to host everything through it for free.

The server that I actually run modmapper on to download all of the mods is a server I already have at home that I also use for other purposes. The output of each run is uploaded to S3, and I also make a full backup of the database and plugin files to Dropbox.

A lot of people thought it was insane that I downloaded every mod⁴, but in reality it wasn't too bad once I got all the issues resolved in modmapper. I just let it run in the background all day and it would chug through the list of mods one-by-one. Most of the time ended up being spent waiting while the Nexus Mod's API hourly rate limit was reached on my account.⁵

As a result of this project I believe I now have the most complete set of all Skyrim plugins to date (extracted plugins only without other textures, models, etc.)⁶. Compressed, it totals around 99 GB, uncompressed: 191 GB.

After I downloaded Skyrim Classic mods in addition to Skyrim Special Edition, here are some counts from the database:

|:---|:---| | Mods | 113,028 | | Files | 330,487 | | Plugins | 534,831 | | Plugin Cell Edits | 33,464,556 |

Parsing Skyrim plugin files

The Skyrim game engine has a concept of worldspaces which are exterior areas where the player can travel to. The biggest of these being, of course, Skyrim itself (which, in the lore, is a province of the continent of Tamriel on the planet Nirn). Worldspaces are recorded in a plugin file as WRLD records.

Worldspaces are then chunked up into a square grid of cells. The Skyrim worldspace consists of a little over 11,000 square cells. Mods that make a changes to the game world have a record in the plugin (a CELL record) with the cell's X and Y coordinates and a list changes in that cell.

There is some prior art (esplugin, TES5Edit, zedit) of open-source programs that could parse Skyrim plugins and extract this data. However, all of these were too broad for my purpose or relied on the assumption of being run in the context of a load order where the master files of a plugin would also be available. I wanted a program that could take a single plugin in isolation and skip through all of the non-relevant parts of it and dump just the CELL and WRLD record data plus some metadata about the plugin from the header as fast as possible.

After discovering the wonderful documentation on the UESP wiki about the Skyrim mod file format, I realized this would be something that would be possible to make myself. skyrim-cell-dump is a Rust library/CLI program that accepts a Skyrim mod file and spits out the header metadata of the plugin, the worlds edited/created, and all of the cells it edits/creates.

Under the hood, it uses the nom crate to read through the plugin until it finds the relevant records, then uses flate2 to decompress any compressed record data, and finally outputs the extracted data formatted to JSON with serde.

Overall, I was pretty happy with this toolkit of tools and was able to quickly get the data I needed from plugins. My only gripe was that I never quite figured out how to properly do error handling with nom. If there was ever an error, I didn't get much data in the error about what failed besides what function it failed in. I often had to resort to peppering in a dozen dbg!() statements to figure out what went wrong.

I built it as both a library and binary crate so that I could import it in other libraries and get the extracted data directly as Rust structs without needing to go through JSON. I'll go more into why this was useful later.

Building the website

Since I wanted to keep server costs low and wanted the site to be as fast as possible for users, I decided pretty early on that the site would be purely static HTML and JavaScript with no backend server. I decided to use the Next.js web framework with TypeScript since it was what I was familiar with using in my day job. While it does have server-side rendering support which would require running a backend Node.js server, it also supports a limited feature-set that can be exported as static HTML.

I host the site on Cloudflare pages which is available on their free tier and made deploying from Github commits a breeze⁷. The web code is in my modmapper-web repo.

The most prominent feature of the website is the interactive satellite map of Skyrim. Two essential resources made this map possible: the map tile images from the UESP skyrim map and Mapbox.

Mapbox provides a JS library for its WebGL map which allows specifying a raster tile source⁸.

The UESP team painstakingly loaded every cell in the Skyrim worldspace in the Creation Kit and took a screenshot. Once I figured out which image tiles mapped to which in-game cell it was relatively easy to put a map together by plugging them into the Mapbox map as a raster tile source.

The heatmap overlaid on the map is created using a Mapbox layer that fills a cell with a color on a gradient from green to red depending on how many edits that cell has across the whole database of mods.

The sidebar on the site is created using React and Redux and uses the next/router to keep track of which page the user is on with URL parameters.

The mod search is implemented using MiniSearch that asynchronously loads the giant search indices for each game containing every mod name and id.

One of the newest features of the site allows users to drill down to a particular plugin within a file of a mod and "Add" it to their list. All of the added plugins will be listed in the sidebar and the cells they edit displayed in purple outlines and conflicts between them displayed in red outlines.

Loading plugins client-side with WebAssembly

A feature that many users requested after the initial release was being able to load a list of the mods currently installed on their game and see which ones of that set conflict with each other⁹. Implementing this feature was one of the most interesting parts of the project. Choosing to use Rust made made it possible, since everything I was running server-side to extract the plugin data could also be done client-side in the browser with the same Rust code compiled to WebAssembly.

I used wasm-pack to create skyrim-cell-dump-wasm which exported the parse_plugin function from my skyrim-cell-dump Rust library compiled to WebAssembly. It also exports a hash_plugin function that creates a unique hash for a plugin file's slice of bytes using seahash so the site can link plugins a user has downloaded on their hard-drive to plugins that have been downloaded by modmapper and saved in the database.

Dragging-and-dropping the Skyrim Data folder on to the webpage or selecting the folder in the "Open Skyrim Data directory" dialog kicks off a process that starts parsing all of the plugin files in that directory in parallel using Web Workers.

I developed my own WorkerPool that manages creating a pool of available workers and assigns them to plugins to process. The pool size is the number of cores on the user's device so that the site can process as many plugins in parallel as possible. After a plugin finishes processing a plugin and sends the output to the redux store, it gets added back to the pool and is then assigned a new plugin to process if there are any¹⁰.

Once all plugins have been loaded, the map updates by displaying all of the cells edited in a purple box and any cells that are edited by more than one plugin in a red box.

Users can also drag-and-drop or paste their plugins.txt file, which is the file that the game uses to define the load order of plugins and which plugins are enabled or disabled. Adding the plugins.txt sorts the list of loaded plugins in the sidebar in load order and enables or disables plugins as defined in the plugins.txt.

Selecting a cell in the map will display all of the loaded cells that edit that cell in the sidebar.

The ability to load plugins straight from a user's hard-drive allows users to map mods that haven't even been uploaded to Nexus Mods.

Vortex integration

The initial mod I released on the Skyrim Special Edition page of Nexus Mods was taken down by the site admins since it didn't contain an actual mod and they didn't agree that it qualified as a "Utility".

Determined to have an actual mod page for Modmapper on Nexus Mods, I decided to make a Vortex integration for modmapper. Vortex is a mod manager made by the developers of Nexus Mods and they allow creating extensions to the tool and have their own mod section for Vortex extensions.

With the help of Pickysaurus, one of the community managers for Nexus Mods, I created a Vortex integration for Modmapper. It adds a context menu option on mods to view the mod in Modmapper with all of the cells it edits selected in purple. It also adds a button next to every plugin file to view just that plugin in Modmapper (assuming it has been processed by Modmapper).

To enable the latter part, I had to include skyrim-cell-dump-wasm in the extension so that I could hash the plugin contents with seahash to get the same hash that Modmapper would have generated. It only does this hashing when you click the "See on Modmapper" button to save from excessive CPU usage when viewing the plugin list.

After releasing the Vortex plugin, Pickysaurus published a news article about modmapper to the Skyrim Special Edition site which also got a lot of nice comments ❤️.

Finishing the collection by adding all Skyrim Classic mods

Skyrim is very silly in that it has many editions. But there was only one that split the modding universe into two: Skyrim Special Edition (SE). It was released in October 2016 with a revamped game engine that brought some sorely needed graphical upgrades. However, it also contained changes to how mods worked, requiring all mod authors to convert their mods to SE. This created big chasm in the library of mods, and Nexus Mods had to make a separate section for SE-only mods.

When I started downloading mods in 2021, I started only with Skyrim SE mods, which, at the time of writing, totals at over 55,000 mods on Nexus Mods.

After releasing with just SE mods, many users requested that all of the classic pre-SE Skyrim mods be added as well. This month, I finally finished downloading all Skyrim Classic mods, which, at the time of writing, totals at over 68,000 mods on Nexus Mods. That brings the total downloaded and processed mods for Modmapper at over 113,000 mods⁴!

The future

A lot of users had great feedback and suggestions on what to add to the site. I could only implement so many of them, though. The rest I've been keeping track of on this Trello board.

Some of the headline items on it are:

• Add Solstheim map

  Since map tiles images are available for that worldspace and because I have already recorded edits to the worldspace in my database, it shouldn't be too terribly difficult.

• Add Mod Organizer 2 plugin

  Lots of people requested this since it's a very popular mod manager compared to Vortex. MO2 supports python extensions so I created skyrim-cell-dump-py to export the Rust plugin processing code to a Python library. I got a bit stuck on actually creating the plugin though, so it might be a while until I get to that.

• Find a way to display interior cell edits on the map

  The map is currently missing edits to interior cells. Since almost all interior cells in Skyrim have a link to the exterior world through a door teleporter, it should be possible to map an interior cell edit to an exterior cell on the map based on which cell the door leads out to.

  That will require digging much more into the plugin files for more data, and city worldspaces will complicate things further. Then there's the question of interiors with multiple doors to different exterior cells, or interior cells nested recursively deep within many other interior cells.

• Create a standalone Electron app that can run outside the browser

  I think this would solve a lot of the issues I ran into while developing the website. Since Electron has a Node.js process running on the user's computer outside the sandboxed browser process, it gives me much more flexibility. It could do things like automatically load a user's plugin files. Or just load plugins at all wihtout having to deal with the annoying dialog that lies to the user saying they're about to upload their entire Data folder hundreds of gigabytes full of files to a server (I really wish the HTMLInputElement.webkitdirectory API would use the same underlying code as the HTML Drag and Drop API which is a lot better).

• Improving the search

  The mod search feature struggles the most with the static generated nature of the site. I found it very hard to pack all of the necessary info for the search index for all 100k+ mods (index for both SE and LE is around 6 MB). Asynchronously loading the indices with MiniSearch keeps it from freezing up the browser, but it does take a very long time to fully load. I can't help think that there's a better way to shard the indices somehow and only fetch what I need based on what the user is typing into the search.

To be clear, a lot of the Todos on the board are pipe-dreams. I may never get to them. This project is sustained purely by my motivation and self-interests and if something is too much of a pain to get working I'll just drop it.

There will also be future Elder Scrolls games, and future Bethesda games based on roughly the same game engine. It would be neat to create similar database for those games as the modding community develops in realtime.

Overall, I'm glad I made something of use to the modding community. I hope to keep the site running for as long as people are modding Skyrim (until the heat-death of the universe, probably).

---

Footnotes

---

1. Unfortunately, I basically lost interest on the mod after working on Modmapper. I might still write a blog post about it eventually since I did a lot of interesting hacking on the Skyrim game engine to try to add some asynchronous multiplayer aspects. Project is here if anyone is curious in the meantime. ↩︎

2. I sadly only have screenshots for some of the comments on that mod since it was eventually taken down by the Nexus Mod admins. See explanation about that in the Vortex integration section. ↩︎

3. As recorded by Cloudflare's server side analytics, which may record a fair amount of bot traffic. I suspect this is the most accurate number I can get since most of my users probably use an ad blocker that blocks client-side analytics. ↩︎

4. Every mod on Nexus Mods except for adult mods since the site restricts viewing adult mods to only logged-in users and I wasn't able to get my scraping bot to log in as a user. ↩︎

5. Apparently my mass-downloading did not go unnoticed by the Nexus Mod admins. I think it's technically against their terms of service to automatically download mods, but I somehow got on their good side and was spared the ban-hammer. I don't recommend anyone else run modmapper themselves on the entire site unless you talk to the admins beforehand and get the okay from them. ↩︎

6. If you would like access to this dataset of plugins to do some data-mining please reach out to me at tyler@hallada.net (Note: only contains plugins files, no models, textures, audio, etc.). I don't plan on releasing it publicly since that would surely go against many mod authors' wishes/licenses. ↩︎

7. I'm not sure I want to recommend anyone else use Cloudflare after the whole Kiwi Farms debacle. I now regret having invested so much of the infrastructure in them. However, I'm only using their free-tier, so at least I am a net-negative for their business? I would recommend others look into Netlify or fastly for similar offerings to Cloudflare pages/CDN. ↩︎

8. I also tried to add a raster Terrain-DEM source for rendering the terrain in 3D. I got fairly far generating my own DEM RGB tiles from an upscaled greyscale heightmap constructed from the LAND records in Skyrim.esm (view it here). But, it came out all wrong: giant cliffs in the middle of the map and tiny spiky lumps with big jumps in elevation at cell boundaries. Seemed like too much work to get right than it was worth it. ↩︎

9. This was the announcement I posted to /r/skyrimmods for this feature ↩︎

10. At first, I noticed a strange issue with re-using the same worker on different plugins multiple times. After a while (~30 reuses per worker), the processing would slow to a crawl and eventually strange things started happening (I was listening to music in my browser and it started to pop and crack). It seemed like the speed of processing increased exponentially to the number of times the worker was reused. So, to avoid this, I had to make the worker pool terminate and recreate workers after every plugin processed. This ended up not being as slow as it sounds and worked fine. However, I recently discovered that wee_alloc, the most suggested allocator to use with rust in wasm, has a memory leak and is mostly unmaintained now. I switched to the default allocator and I didn't run into the exponentially slow re-use problem. For some reason, the first run on a fresh tab is always much faster than the second run, but subsequent runs are still fairly stable in processing time. ↩︎