Compare commits
No commits in common. "master" and "options" have entirely different histories.
@ -1 +0,0 @@
|
||||
3.1.2
|
||||
@ -1,54 +0,0 @@
|
||||
==================================================================
|
||||
https://keybase.io/thallada
|
||||
--------------------------------------------------------------------
|
||||
|
||||
I hereby claim:
|
||||
|
||||
* I am an admin of http://www.hallada.net
|
||||
* I am thallada (https://keybase.io/thallada) on keybase.
|
||||
* I have a public key ASDOaZj16vBUa4vjFa4dhC7map8qIX5MUSCqjgWeX1CfbQo
|
||||
|
||||
To do so, I am signing this object:
|
||||
|
||||
{
|
||||
"body": {
|
||||
"key": {
|
||||
"eldest_kid": "0120ce6998f5eaf0546b8be315ae1d842ee66a9f2a217e4c5120aa8e059e5f509f6d0a",
|
||||
"host": "keybase.io",
|
||||
"kid": "0120ce6998f5eaf0546b8be315ae1d842ee66a9f2a217e4c5120aa8e059e5f509f6d0a",
|
||||
"uid": "bf2238122821bbc309d5bf1ed2421d19",
|
||||
"username": "thallada"
|
||||
},
|
||||
"service": {
|
||||
"hostname": "www.hallada.net",
|
||||
"protocol": "http:"
|
||||
},
|
||||
"type": "web_service_binding",
|
||||
"version": 1
|
||||
},
|
||||
"client": {
|
||||
"name": "keybase.io go client",
|
||||
"version": "1.0.18"
|
||||
},
|
||||
"ctime": 1486587579,
|
||||
"expire_in": 504576000,
|
||||
"merkle_root": {
|
||||
"ctime": 1486587569,
|
||||
"hash": "099473519e50871bbe05a57da09e5ba8fa575e8a51e259dd6ad6e8e4ead07fdb63895424659bc01aef600d5caa2ddaf32bd194fec6e08b54f623c0bf381df30c",
|
||||
"seqno": 846152
|
||||
},
|
||||
"prev": "7ec1319be81dde8362b777b669904944eaebd592206c17953c628749b41cf992",
|
||||
"seqno": 9,
|
||||
"tag": "signature"
|
||||
}
|
||||
|
||||
which yields the signature:
|
||||
|
||||
hKRib2R5hqhkZXRhY2hlZMOpaGFzaF90eXBlCqNrZXnEIwEgzmmY9erwVGuL4xWuHYQu5mqfKiF+TFEgqo4Fnl9Qn20Kp3BheWxvYWTFAvZ7ImJvZHkiOnsia2V5Ijp7ImVsZGVzdF9raWQiOiIwMTIwY2U2OTk4ZjVlYWYwNTQ2YjhiZTMxNWFlMWQ4NDJlZTY2YTlmMmEyMTdlNGM1MTIwYWE4ZTA1OWU1ZjUwOWY2ZDBhIiwiaG9zdCI6ImtleWJhc2UuaW8iLCJraWQiOiIwMTIwY2U2OTk4ZjVlYWYwNTQ2YjhiZTMxNWFlMWQ4NDJlZTY2YTlmMmEyMTdlNGM1MTIwYWE4ZTA1OWU1ZjUwOWY2ZDBhIiwidWlkIjoiYmYyMjM4MTIyODIxYmJjMzA5ZDViZjFlZDI0MjFkMTkiLCJ1c2VybmFtZSI6InRoYWxsYWRhIn0sInNlcnZpY2UiOnsiaG9zdG5hbWUiOiJ3d3cuaGFsbGFkYS5uZXQiLCJwcm90b2NvbCI6Imh0dHA6In0sInR5cGUiOiJ3ZWJfc2VydmljZV9iaW5kaW5nIiwidmVyc2lvbiI6MX0sImNsaWVudCI6eyJuYW1lIjoia2V5YmFzZS5pbyBnbyBjbGllbnQiLCJ2ZXJzaW9uIjoiMS4wLjE4In0sImN0aW1lIjoxNDg2NTg3NTc5LCJleHBpcmVfaW4iOjUwNDU3NjAwMCwibWVya2xlX3Jvb3QiOnsiY3RpbWUiOjE0ODY1ODc1NjksImhhc2giOiIwOTk0NzM1MTllNTA4NzFiYmUwNWE1N2RhMDllNWJhOGZhNTc1ZThhNTFlMjU5ZGQ2YWQ2ZThlNGVhZDA3ZmRiNjM4OTU0MjQ2NTliYzAxYWVmNjAwZDVjYWEyZGRhZjMyYmQxOTRmZWM2ZTA4YjU0ZjYyM2MwYmYzODFkZjMwYyIsInNlcW5vIjo4NDYxNTJ9LCJwcmV2IjoiN2VjMTMxOWJlODFkZGU4MzYyYjc3N2I2Njk5MDQ5NDRlYWViZDU5MjIwNmMxNzk1M2M2Mjg3NDliNDFjZjk5MiIsInNlcW5vIjo5LCJ0YWciOiJzaWduYXR1cmUifaNzaWfEQIiv6L2Js0MEXpgiHcIhP9B3MmBl+81QA0z32QYT5XWXsH/6rsylYYQCLWjrIXAILrOIoH5Jyd2GFfpU+O8A/w2oc2lnX3R5cGUgpGhhc2iCpHR5cGUIpXZhbHVlxCDXYP2O217NRz/lX6Q6G54D0G6/EIfX2OJY/hduqsrYNqN0YWfNAgKndmVyc2lvbgE=
|
||||
|
||||
And finally, I am proving ownership of this host by posting or
|
||||
appending to this document.
|
||||
|
||||
View my publicly-auditable identity here: https://keybase.io/thallada
|
||||
|
||||
==================================================================
|
||||
@ -1,3 +0,0 @@
|
||||
{
|
||||
"m.server": "synapse.hallada.net:443"
|
||||
}
|
||||
4
Gemfile
4
Gemfile
@ -1,4 +1,2 @@
|
||||
source 'http://rubygems.org'
|
||||
|
||||
gem "webrick"
|
||||
gem 'github-pages', group: :jekyll_plugins
|
||||
gem 'github-pages'
|
||||
|
||||
55
README.md
55
README.md
@ -4,21 +4,9 @@ thallada.github.io
|
||||
This is the latest version of my personal website. It is a static website built
|
||||
with [Jekyll](http://jekyllrb.com/).
|
||||
|
||||
See it at [https://www.hallada.net/](https://www.hallada.net/).
|
||||
|
||||
## Build Locally
|
||||
|
||||
To run a version of the site in development locally. Checkout this repo and
|
||||
then:
|
||||
|
||||
1. `cd thallada.github.io`
|
||||
2. [Install Jekyll](https://jekyllrb.com/docs/installation/)
|
||||
3. Run `bundle install`
|
||||
3. Run `bundle exec jekyll serve`
|
||||
4. Visit `http://localhost:4000` to view the website
|
||||
|
||||
## Magic
|
||||
See it at [http://www.hallada.net/](http://www.hallada.net/).
|
||||
|
||||
##Magic##
|
||||
Most of the development work of this website went into creating what I like to
|
||||
call "magic", or the dynamic background to my homepage. A few seconds after
|
||||
loading the page, a branching web of colored tendrils will grow in a random
|
||||
@ -41,14 +29,11 @@ random, colorful, and more CPU efficient.
|
||||
It was really fun to tweak various variables in the script and see how the
|
||||
animation reacted. It didn't take much tweaking to get the lines to appear like
|
||||
lightning flashing in the distant background, or like cracks splitting the
|
||||
screen, or like growing forest of sprouting trees.
|
||||
|
||||
You can play around with these variables yourself on the [/magic
|
||||
page](https://www.hallada.net/magic) which has sliders for tweaking the
|
||||
animations in realtime.
|
||||
|
||||
## Layout & CSS
|
||||
screen, or like growing forest of sprouting trees. A future project may involve
|
||||
putting the magic up on its own webpage and add UI dials to allow anyone to
|
||||
change these variables in realtime.
|
||||
|
||||
##Layout & CSS##
|
||||
I use a [grid system devised by Adam Kaplan](http://www.adamkaplan.me/grid/) and
|
||||
with some pieces from [Jorden Lev](http://jordanlev.github.io/grid/). It is
|
||||
set-up by scratch in my `main.css`. I decided on this so that I would not have
|
||||
@ -120,37 +105,11 @@ desktop, use `hide-desktop` instead.
|
||||
</div>
|
||||
```
|
||||
|
||||
I had an issue with displaying elements on desktop that had the class
|
||||
"hide-mobile", so you can add the following classes to make sure they redisplay
|
||||
in the right display type correctly:
|
||||
|
||||
* `hide-mobile-block`
|
||||
* `hide-mobile-inline-block`
|
||||
* `hide-mobile-inline`
|
||||
* `hide-deskop-block`
|
||||
* `hide-desktop-inline-block`
|
||||
* `hide-desktop-inline`
|
||||
|
||||
I could add more for each `display` property, but I'm trying to find a better
|
||||
way of fixing this without adding these second classes.
|
||||
|
||||
Another note: I use [box-sizing (as suggested by Paul
|
||||
Irish)](http://www.paulirish.com/2012/box-sizing-border-box-ftw/), which I think
|
||||
makes dealing with sizing elements a lot more sane.
|
||||
|
||||
### Light & Dark Themes
|
||||
|
||||
In 2020, I created a dark theme for the website. The dark theme is used if it
|
||||
detects that the user's OS is set to prefer a dark theme [using the
|
||||
`prefers-color-scheme` `@media`
|
||||
query](https://css-tricks.com/dark-modes-with-css/).
|
||||
|
||||
To allow the user to select a theme separate from their OS's theme, I have also
|
||||
included [a switch that can toggle between the two
|
||||
themes](https://github.com/GoogleChromeLabs/dark-mode-toggle).
|
||||
|
||||
## Attributions
|
||||
|
||||
##Attributions##
|
||||
[Book](http://thenounproject.com/term/book/23611/) designed by [Nherwin
|
||||
Ardoña](http://thenounproject.com/nherwinma) from the Noun Project.
|
||||
|
||||
|
||||
29
_config.yml
29
_config.yml
@ -1,32 +1,9 @@
|
||||
title: Tyler Hallada
|
||||
name: Tyler Hallada - Blog
|
||||
description: Musings on technology, literature, and interesting topics
|
||||
author: thallada
|
||||
url: https://www.hallada.net
|
||||
blog_url: https://www.hallada.net/blog
|
||||
assets: https://hallada.net/assets/
|
||||
logo: /img/profile_icon_128x128.jpg
|
||||
social:
|
||||
name: Tyler Hallada
|
||||
links:
|
||||
- https://twitter.com/tyhallada
|
||||
- https://www.facebook.com/tyhallada
|
||||
- https://www.linkedin.com/in/thallada/
|
||||
- https://github.com/thallada
|
||||
defaults:
|
||||
-
|
||||
scope:
|
||||
path: ""
|
||||
values:
|
||||
image: /img/profile_icon_300x200.jpg
|
||||
markdown: kramdown
|
||||
kramdown:
|
||||
syntax_highlighter: rouge
|
||||
excerpt_separator: "<!--excerpt-->"
|
||||
url: http://hallada.net/blog
|
||||
markdown: redcarpet
|
||||
pygments: true
|
||||
paginate: 10
|
||||
paginate_path: "blog/page:num"
|
||||
gems:
|
||||
- jekyll-redirect-from
|
||||
- jekyll-paginate
|
||||
- jekyll-seo-tag
|
||||
include: [".well-known"]
|
||||
|
||||
@ -1,3 +0,0 @@
|
||||
thallada:
|
||||
picture: /img/profile_icon_128x128.jpg
|
||||
twitter: tyhallada
|
||||
@ -1,12 +0,0 @@
|
||||
<div class="card">
|
||||
<div class="row clearfix">
|
||||
<div class="column full">
|
||||
<script data-isso="https://comments.hallada.net/"
|
||||
src="https://comments.hallada.net/js/embed.min.js"></script>
|
||||
|
||||
<section id="isso-thread">
|
||||
<noscript>Javascript needs to be activated to view comments.</noscript>
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -1,36 +0,0 @@
|
||||
<div class="card">
|
||||
<div class="subscribe-form">
|
||||
<div class="row clearfix">
|
||||
<div class="column full">
|
||||
<h3>Subscribe to my future posts</h3>
|
||||
</div>
|
||||
</div>
|
||||
<form action="https://list.hallada.net/subscribe" method="POST" accept-charset="utf-8">
|
||||
<div class="row clearfix">
|
||||
<div class="column half">
|
||||
<label for="name">Name (optional)</label><br />
|
||||
<input type="text" name="name" id="name" />
|
||||
</div>
|
||||
<div class="column half">
|
||||
<label for="email">Email</label><br />
|
||||
<input type="email" name="email" id="email" />
|
||||
</div>
|
||||
</div>
|
||||
<div class="row clearfix">
|
||||
<div style="display:none;">
|
||||
<label for="hp">HP</label><br />
|
||||
<input type="text" name="hp" id="hp" />
|
||||
</div>
|
||||
<input type="hidden" name="list" value="Q7JrUBzeCeftZqwDtxsQ9w" />
|
||||
<div class="column half">
|
||||
<input type="submit" name="submit" id="submit" value="Submit" />
|
||||
</div>
|
||||
<div class="column half">
|
||||
<span class="form-rss">Or subscribe to my <a href="/feed.xml">RSS feed</a></span>
|
||||
</div>
|
||||
</div>
|
||||
</form>
|
||||
<div class="row clearfix">
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -1,5 +1,5 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
|
||||
@ -10,62 +10,54 @@
|
||||
<link rel="stylesheet" href="/css/normalize.css">
|
||||
<!-- Fix IE -->
|
||||
<!--[if lt IE 9]>
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7/html5shiv.js"></script>
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/respond.js/1.4.2/respond.js"></script>
|
||||
<script src="http://cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7/html5shiv.js"></script>
|
||||
<script src="http://cdnjs.cloudflare.com/ajax/libs/respond.js/1.4.2/respond.js"></script>
|
||||
<![endif]-->
|
||||
|
||||
<!-- syntax highlighting CSS -->
|
||||
<link rel="stylesheet" href="/css/syntax.css">
|
||||
<link rel="stylesheet" href="/css/syntax_dark.css" media="(prefers-color-scheme: dark)">
|
||||
|
||||
<!-- Custom CSS -->
|
||||
<link rel="stylesheet" href="/css/main.css">
|
||||
<link rel="stylesheet" href="/css/main_dark.css" media="(prefers-color-scheme: dark)">
|
||||
|
||||
<!-- Web Fonts -->
|
||||
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,300,700" rel="stylesheet" type="text/css">
|
||||
<link href='http://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,300,700' rel='stylesheet' type='text/css'>
|
||||
|
||||
<!-- RSS Feed -->
|
||||
<link href="/feed.xml" rel="alternate" type="application/atom+xml">
|
||||
<link href='/feed.xml' rel='alternate' type='application/atom+xml'>
|
||||
|
||||
<!-- Favicon -->
|
||||
<link rel="shortcut icon" href="/favicon.png">
|
||||
<link rel="shortcut icon" href="/favicon.png" />
|
||||
|
||||
<!-- Cloudflare Web Analytics -->
|
||||
<script defer src='https://static.cloudflareinsights.com/beacon.min.js' data-cf-beacon='{"token": "54df1dc81a2d4cb7920b456212bbd437"}'></script>
|
||||
<!-- End Cloudflare Web Analytics -->
|
||||
<!-- Google Analytics -->
|
||||
<script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
|
||||
|
||||
<script type="module" src="https://unpkg.com/dark-mode-toggle"></script>
|
||||
ga('create', 'UA-39880341-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
|
||||
{% seo %}
|
||||
</script>
|
||||
<!-- End Google Analytics -->
|
||||
</head>
|
||||
<body>
|
||||
<div class="root">
|
||||
<div class="container">
|
||||
<div class="row clearfix header">
|
||||
<h1 class="title"><a href="/blog/">{{ site.name }}</a></h1>
|
||||
<a class="extra" href="/">home</a>
|
||||
</div>
|
||||
{{ content }}
|
||||
<div class="row clearfix rss">
|
||||
<a class="rss" href="/feed.xml"><img src="/img/rss.png" alt="RSS" class="icon" /></a>
|
||||
</div>
|
||||
<div class="row clearfix footer">
|
||||
<div class="column full contact">
|
||||
<p class="contact-info">
|
||||
<a href="mailto:tyler@hallada.net">tyler@hallada.net</a>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="container">
|
||||
<div class="row clearfix header">
|
||||
<h1 class="title"><a href="/blog">{{ site.name }}</a></h1>
|
||||
<a class="extra" href="/">home</a>
|
||||
</div>
|
||||
<div class="theme-toggle">
|
||||
<dark-mode-toggle
|
||||
id="dark-mode-toggle-1"
|
||||
appearance="toggle"
|
||||
light="Dark"
|
||||
dark="Light"
|
||||
permanent
|
||||
></dark-mode-toggle>
|
||||
{{ content }}
|
||||
<div class="row clearfix rss">
|
||||
<a class="rss" href="/feed.xml"><img src="/img/rss.png" alt="RSS"/></a>
|
||||
</div>
|
||||
<div class="row clearfix footer">
|
||||
<div class="column full contact">
|
||||
<p class="contact-info">
|
||||
<a href="mailto:tyler@hallada.net">tyler@hallada.net</a>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
|
||||
@ -10,57 +10,49 @@
|
||||
<link rel="stylesheet" href="/css/normalize.css">
|
||||
<!-- Fix IE -->
|
||||
<!--[if lt IE 9]>
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7/html5shiv.js"></script>
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/respond.js/1.4.2/respond.js"></script>
|
||||
<script src="http://cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7/html5shiv.js"></script>
|
||||
<script src="http://cdnjs.cloudflare.com/ajax/libs/respond.js/1.4.2/respond.js"></script>
|
||||
<![endif]-->
|
||||
|
||||
<!-- syntax highlighting CSS -->
|
||||
<link rel="stylesheet" href="/css/syntax.css">
|
||||
<link rel="stylesheet" href="/css/syntax_dark.css" media="(prefers-color-scheme: dark)">
|
||||
|
||||
<!-- Custom CSS -->
|
||||
<link rel="stylesheet" href="/css/main.css">
|
||||
<link rel="stylesheet" href="/css/main_dark.css" media="(prefers-color-scheme: dark)">
|
||||
|
||||
<!-- RSS Feed -->
|
||||
<link href="/feed.xml" rel="alternate" type="application/atom+xml">
|
||||
<link href='/feed.xml' rel='alternate' type='application/atom+xml'>
|
||||
|
||||
<!-- Web Fonts -->
|
||||
<link href="https://fonts.googleapis.com/css?family=Open+Sans:400,400italics,300,300italics,200" rel="stylesheet" type="text/css">
|
||||
<link href='http://fonts.googleapis.com/css?family=Open+Sans:400,400italics,300,300italics,200' rel='stylesheet' type='text/css'>
|
||||
|
||||
<!-- Icon Fonts -->
|
||||
<link rel="stylesheet" href="/css/ionicons.min.css">
|
||||
|
||||
<!-- Favicon -->
|
||||
<link rel="shortcut icon" href="/favicon.png">
|
||||
<link rel="shortcut icon" href="/favicon.png" />
|
||||
|
||||
<!-- Scripts -->
|
||||
<script src="/js/AnimationFrame.min.js"></script>
|
||||
<script async src="/js/magic.js"></script>
|
||||
|
||||
<!-- Cloudflare Web Analytics -->
|
||||
<script defer src='https://static.cloudflareinsights.com/beacon.min.js' data-cf-beacon='{"token": "54df1dc81a2d4cb7920b456212bbd437"}'></script>
|
||||
<!-- End Cloudflare Web Analytics -->
|
||||
<!-- Google Analytics -->
|
||||
<script>
|
||||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||||
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
|
||||
|
||||
<script type="module" src="https://unpkg.com/dark-mode-toggle"></script>
|
||||
ga('create', 'UA-39880341-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
|
||||
{% seo %}
|
||||
</script>
|
||||
<!-- End Google Analytics -->
|
||||
</head>
|
||||
<body>
|
||||
<div class="root">
|
||||
<canvas id="magic"></canvas>
|
||||
<div class="container">
|
||||
{{ content }}
|
||||
</div>
|
||||
<div class="theme-toggle">
|
||||
<dark-mode-toggle
|
||||
id="dark-mode-toggle-1"
|
||||
appearance="toggle"
|
||||
light="Dark"
|
||||
dark="Light"
|
||||
permanent
|
||||
></dark-mode-toggle>
|
||||
</div>
|
||||
<canvas id="magic"></canvas>
|
||||
<div class="container">
|
||||
{{ content }}
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@ -1,20 +1,18 @@
|
||||
---
|
||||
layout: default
|
||||
---
|
||||
|
||||
<div class="card">
|
||||
<div class="row clearfix">
|
||||
<div class="column full post-header">
|
||||
<h2 class="post-title"><a href="{{ post.url }}">{{ page.title }}</a></h2>
|
||||
<div class="row clearfix post-header">
|
||||
<div class="column three-fourths">
|
||||
<a href="{{ post.url }}"><h2 class="post-title">{{ page.title }}</h2></a>
|
||||
</div>
|
||||
<div class="column fourth">
|
||||
<span class="timestamp">{{ page.date | date_to_string }}</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="row clearfix">
|
||||
<div class="column full post">{{ content }}</div>
|
||||
<div class="column full post">
|
||||
{{ content }}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% include comments.html %}
|
||||
|
||||
<!-- disabling until I fix the mail form -->
|
||||
<!-- {% include mail-form.html %} -->
|
||||
|
||||
@ -13,7 +13,6 @@ and was pretty familiar with it and I was beginning to get familiar with
|
||||
was what I was working with at [Valti](https://www.valti.com), and I was really
|
||||
liking making websites with it. It took what made Python awesome and applied
|
||||
that to web development.
|
||||
<!--excerpt-->
|
||||
|
||||
I started from a blank Django project and built it up from there. Django's
|
||||
Object-Relational Mapper (ORM) can be boiled down to this: python classes
|
||||
|
||||
@ -11,7 +11,6 @@ you can tell, there hasn't been any posts since my first ["Hello,
|
||||
World!"](/2012/12/03/hello-world) post. Sure, I've been working on projects, but I
|
||||
just haven't gotten to the point in any of those projects where I felt like I
|
||||
could blog in detail about it.
|
||||
<!--excerpt-->
|
||||
|
||||
Then I watched this great talk that [Brian
|
||||
Jones](http://pyvideo.org/speaker/352/brian-k-jones) gave at
|
||||
@ -19,7 +18,7 @@ Jones](http://pyvideo.org/speaker/352/brian-k-jones) gave at
|
||||
pointed out to me:
|
||||
|
||||
<div class="video-container"><iframe width="640" height="360"
|
||||
src="https://www.youtube.com/embed/BBfW3m3TK0w?feature=player_embedded"
|
||||
src="http://www.youtube.com/embed/BBfW3m3TK0w?feature=player_embedded"
|
||||
frameborder="0" allowfullscreen></iframe></div>
|
||||
|
||||
One point that he makes that really resonates with me is how I should write
|
||||
|
||||
@ -11,7 +11,6 @@ keeps track of the status of every machine and displays it on a
|
||||
[website](http://gmu.esuds.net/) so students can check how full the machines
|
||||
are before making the trek down to the laundry rooms. The system emails each
|
||||
student when their laundry is finished as well.
|
||||
<!--excerpt-->
|
||||
|
||||
The only problem is that their user interface is pretty atrocious. I wrote up a
|
||||
[usability analysis](https://gist.github.com/thallada/5351114) of the site for
|
||||
@ -27,7 +26,7 @@ which made it easy for me to dive right in. I'll probably try out
|
||||
[d3js](http://d3js.org/) for my next visualization project though, it looks a
|
||||
whole lot more advanced.
|
||||
|
||||
### Current laundry usage charts
|
||||
###Current laundry usage charts###
|
||||
|
||||
I created an [app](/laundry) in [Django](https://www.djangoproject.com/) to
|
||||
display current laundry machine usage charts for all of the laundry rooms on
|
||||
@ -48,7 +47,7 @@ folder).
|
||||
The point was to make this as dead simple and easy to use as possible. Do you
|
||||
think I succeeded?
|
||||
|
||||
### Weekly laundry usage chart
|
||||
###Weekly laundry usage chart###
|
||||
|
||||
Knowing the *current* laundry machine usage is nice for saving a wasted trip
|
||||
down to the laundry room, but what if you wanted to plan ahead and do your
|
||||
|
||||
@ -12,7 +12,6 @@ streamed to the user's Flash player (in their browser) bit-by-bit, the full
|
||||
video file is never given to the user for them to keep. This is desirable to a
|
||||
lot of media companies because then they can force you to watch through ads to
|
||||
see their content and can charge you to download the full video.
|
||||
<!--excerpt-->
|
||||
|
||||
However, [RTMPDump](http://rtmpdump.mplayerhq.hu/), an open-source tool
|
||||
designed to intercept RTMP streams, can download the full video.
|
||||
@ -28,7 +27,7 @@ Since this is questionably legal, make sure you understand any Terms of
|
||||
Services you accepted or laws in your locality regarding this before you follow
|
||||
the steps below ;).
|
||||
|
||||
### Have Linux
|
||||
###Have Linux###
|
||||
|
||||
Most of these instructions will assume you have Ubuntu, but
|
||||
most distributions will work.
|
||||
@ -37,18 +36,18 @@ While RTMPDump works on a variety of operating systems, I've only researched
|
||||
how to do this on Linux. Feel free to comment if you know how to do this in
|
||||
Windows or OSX.
|
||||
|
||||
### Install RTMPDump
|
||||
###Install RTMPDump###
|
||||
|
||||
This open source goodness can be found at
|
||||
[http://rtmpdump.mplayerhq.hu/](http://rtmpdump.mplayerhq.hu/) or you can just
|
||||
intall it using your Linux distro's package manager. For Ubuntu, that would be
|
||||
typing the following into your terminal:
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
sudo apt-get install rtmpdump
|
||||
~~~
|
||||
```
|
||||
|
||||
### Redirect ALL the RTMP!
|
||||
###Redirect ALL the RTMP!###
|
||||
|
||||
Now we need to configure your firewall to redirect
|
||||
all RTMP traffic to a local port on your computer (Note: this will screw up any
|
||||
@ -56,11 +55,11 @@ RTMP streaming video you try to watch on your computer, so make sure you run
|
||||
the undo command in one of the later steps to return things to normal). Type
|
||||
the following into your terminal, there should be no output from the command:
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
sudo iptables -t nat -A OUTPUT -p tcp --dport 1935 -j REDIRECT
|
||||
~~~
|
||||
```
|
||||
|
||||
### Run rtmpsrv
|
||||
###Run rtmpsrv###
|
||||
|
||||
When you install `rtmpdump`, a program called `rtmpsrv`
|
||||
should have been bundled with it and installed as well. We will want to run
|
||||
@ -74,7 +73,7 @@ This should output something that looks like this:
|
||||
|
||||
Streaming on rtmp://0.0.0.0:1935
|
||||
|
||||
### Feed rtmpsrv the Precious Video
|
||||
###Feed rtmpsrv the Precious Video###
|
||||
|
||||
Now go to your browser and open/refresh
|
||||
the page with the desired video. Try playing the video. If nothing happens and
|
||||
@ -88,24 +87,24 @@ will need it later.
|
||||
|
||||
You can CTRL+C out of rtmpsrv now that we have what we need.
|
||||
|
||||
### Undo the Redirection
|
||||
###Undo the Redirection###
|
||||
|
||||
You must undo the iptables redirection command we
|
||||
performed earlier before you can do anything else, so run this in your
|
||||
terminal:
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
sudo iptables -t nat -D OUTPUT -p tcp --dport 1935 -j REDIRECT
|
||||
~~~
|
||||
```
|
||||
|
||||
### Finally, Download the Precious Video
|
||||
###Finally, Download the Precious Video###
|
||||
|
||||
Now paste that command you copied
|
||||
from the rtmpsrv output in the step before last into your terminal prompt and
|
||||
hit enter. You should now see a torrent of `INFO` printout along with a
|
||||
percentage as the video is being downloaded.
|
||||
|
||||
### Feast Eyes on Precious Video
|
||||
###Feast Eyes on Precious Video###
|
||||
|
||||
Once downloaded, the video file, which has a
|
||||
`flv` extension and was named by the `-o` parameter in the command you copied
|
||||
|
||||
@ -11,7 +11,6 @@ met since the past two internships I've had at [Valti](https:/www.valti.com/)
|
||||
and [Humbug](https://humbughq.com/) in Cambridge, Massachusetts. Seeing as it
|
||||
encapsulated what I've learned culturally since then, I decided to post it here
|
||||
as well.*
|
||||
<!--excerpt-->
|
||||
|
||||
Hackers -- not your malicious meddling Hollywood-style speed-typists -- but the
|
||||
type who sees a toaster and turns it into a computer capable of etching emails
|
||||
|
||||
@ -12,7 +12,6 @@ to create a homepage for the University's bookstore website, applying all of the
|
||||
usability principles we had learned over the semester. I ended up working on it
|
||||
when I wanted to procrastinate on assignments in my other classes, so I put
|
||||
quite a bit of effort into it.
|
||||
<!--excerpt-->
|
||||
|
||||
See it here: [swe205.hallada.net](http://swe205.hallada.net)
|
||||
<div style="text-align: center">
|
||||
|
||||
@ -10,7 +10,6 @@ includes redditing. I probably spend far too much time on
|
||||
way to view reddit through the command-line. [w3m](http://w3m.sourceforge.net/)
|
||||
could render reddit okay, but I couldn't view my personal front-page because
|
||||
that required me to login to my profile.
|
||||
<!--excerpt-->
|
||||
|
||||
The solution was [cortex](http://cortex.glacicle.org/), a CLI app for viewing
|
||||
reddit.
|
||||
@ -18,19 +17,19 @@ reddit.
|
||||
However, I kind of got tired of viewing reddit through w3m, the header alone is
|
||||
a few pages long to scroll through, and the CSS for the comments doesn't load so
|
||||
there isn't any sense of threading. But, then I discovered reddit's mobile
|
||||
website: [http://reddit.com/.mobile](http://reddit.com/.mobile), and it looks absolutely
|
||||
website: [http://m.reddit.com](http://m.reddit.com), and it looks absolutely
|
||||
beautiful in w3m. In fact, I think I prefer it to the normal website in any
|
||||
modern browser; there are no distractions, just pure content.
|
||||
|
||||
<a href="/img/blog/w3m_mobile_reddit.png"><img src="/img/blog/w3m_mobile_reddit.png" alt="m.reddit.com rendered in w3m"></a>
|
||||
|
||||
In order to get cortex to open the mobile version of reddit, I made a bash
|
||||
script wrapper around w3m that takes urls and appends `".mobile"` to the end of
|
||||
reddit urls before passing them to w3m (as well as fixing a double forward slash
|
||||
error in the comment uri cortex outputs that desktop reddit accepts but mobile
|
||||
reddit 404s on). The script:
|
||||
script wrapper around w3m that takes urls and replaces `"http://reddit.com"` and
|
||||
`"http://www.reddit.com"` with `"http://m.reddit.com"` before passing them to
|
||||
w3m (as well as fixing a double forward slash error in the comment uri cortex
|
||||
outputs that desktop reddit accepts but mobile reddit 404s on). The script:
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
args=()
|
||||
@ -46,17 +45,8 @@ done
|
||||
args+=("$@")
|
||||
for arg in "${args[@]}" ; do
|
||||
# Switch to mobile reddit
|
||||
url=$arg
|
||||
mobile='.mobile'
|
||||
if [[ $url =~ http:\/\/www.reddit.com || $url =~ http:\/\/reddit.com ]]
|
||||
then
|
||||
if [[ $url =~ \/$ ]]
|
||||
then
|
||||
url=$url$mobile
|
||||
else
|
||||
url=$url'/'$mobile
|
||||
fi
|
||||
fi
|
||||
url=${arg/http:\/\/reddit.com/http:\/\/m.reddit.com}
|
||||
url=${url/http:\/\/www.reddit.com/http:\/\/m.reddit.com}
|
||||
# Fix double backslash error in comment uri for mobile reddit
|
||||
url=${url/\/\/comments/\/comments}
|
||||
if [[ $t == "1" ]]; then
|
||||
@ -65,7 +55,7 @@ for arg in "${args[@]}" ; do
|
||||
w3m "${url}"
|
||||
fi
|
||||
done
|
||||
~~~
|
||||
```
|
||||
|
||||
Since I regurally use [Tmux](http://tmux.sourceforge.net/) (with
|
||||
[Byobu](http://byobu.co/)), I also added an optional `-t`/`--tmux` switch that
|
||||
@ -74,10 +64,10 @@ will open w3m in a temporary new tmux window that will close when w3m is closed.
|
||||
I saved the script as `w3m-reddit` and made it an executable command. In Ubuntu
|
||||
that's done with the following commands:
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
$ sudo mv w3m-reddit /usr/bin/
|
||||
$ sudo chmod +x /usr/bin/w3m-reddit
|
||||
~~~
|
||||
```
|
||||
|
||||
Now cortex needs to be configured to use `w3m-reddit`, and that's done by
|
||||
setting `browser-command` in the cortex config at `~/.cortex/config` to
|
||||
@ -103,9 +93,3 @@ scrapping the whole thing and starting over in Python instead.
|
||||
|
||||
Stay tuned for more posts on how I view images and videos efficiently from the
|
||||
command-line.
|
||||
|
||||
EDIT 04/25/2015: Reddit seems to have gotten rid of their old mobile reddit site
|
||||
and replaced it with a more modern version that unfortunately doesn't look as
|
||||
good in w3m. However, the old mobile site is still accessable by adding a
|
||||
".mobile" to the end of urls. The script above has been edited to reflect this
|
||||
change.
|
||||
|
||||
@ -17,7 +17,6 @@ customizability and compatibility with other programs. There's nothing more
|
||||
powerful than being able to whip up a small python or bash script that interacts
|
||||
with a couple of other programs to achieve something instantly that optimizes my
|
||||
work flow.
|
||||
<!--excerpt-->
|
||||
|
||||
I use the [Awesome](http://awesome.naquadah.org/) window manager, which works
|
||||
great for tiling up terminal windows right up next to browser windows. However,
|
||||
@ -54,7 +53,7 @@ This is how I got it setup (on any Ubuntu machine with sudo privileges):
|
||||
|
||||
Save the following python file in `/usr/bin/` as `search-pane` (no extension):
|
||||
|
||||
~~~ python
|
||||
```python
|
||||
#!/usr/bin/python
|
||||
from subprocess import call, check_output
|
||||
from threading import Thread
|
||||
@ -107,27 +106,27 @@ except Exception, errtxt:
|
||||
print errtxt
|
||||
|
||||
call(['w3m', url]) # pass url off to w3m
|
||||
~~~
|
||||
```
|
||||
|
||||
Make the directory and file for search history:
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
mkdir ~/.search-pane
|
||||
touch ~/.search-pane/history
|
||||
~~~
|
||||
```
|
||||
|
||||
Allow anyone to execute the python script (make it into a program):
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
chmod a+x /usr/bin/search-pane
|
||||
~~~
|
||||
```
|
||||
|
||||
To get quick access to the program from the command-line edit `~/.bashrc` to
|
||||
add:
|
||||
|
||||
~~~ bash
|
||||
```bash
|
||||
alias s='search-pane'
|
||||
~~~
|
||||
```
|
||||
|
||||
To add byobu key bindings edit `~/.byobu/keybindings.tmux` (or `/usr/share/byobu/keybindings/f-keys.tmux`):
|
||||
|
||||
|
||||
@ -9,7 +9,6 @@ just glorified browsers, right? What if I wanted to do anything outside of the
|
||||
browser? Why would you spend [$1299 or $1449 for a
|
||||
computer](https://www.google.com/intl/en/chrome/devices/chromebooks.html#pixel)
|
||||
that can only run a browser?
|
||||
<!--excerpt-->
|
||||
|
||||
While I know a lot of people who buy expensive MacBooks only to just use a web
|
||||
browser and iTunes, I’m a bit more of a power user and I need things like
|
||||
@ -83,7 +82,7 @@ of tweaking. If anyone has read my past posts, they know that I am obsessed
|
||||
with configuring things. Here is what I came up with for everything I would
|
||||
ever need to do on my Chromebook:
|
||||
|
||||
### Writing
|
||||
###Writing###
|
||||
|
||||
I spent a lot of time downloading
|
||||
[various](https://chrome.google.com/webstore/detail/write-space/aimodnlfiikjjnmdchihablmkdeobhad)
|
||||
@ -116,7 +115,7 @@ hassle though, so I often just stick to the default style. It’s a sign that I
|
||||
am procrastinating if I’m trying to look for the “perfect template” to write in
|
||||
anyways.
|
||||
|
||||
### Programming
|
||||
###Programming###
|
||||
|
||||
I’ve gotten so used to [vim](http://www.vim.org/) in a Linux
|
||||
terminal that I don’t think I could ever use any other editor. There are a few
|
||||
@ -151,7 +150,7 @@ have all of the great chrome apps and extensions right at my fingertips.
|
||||
Especially when some apps can be opened up in small panels in the corner of the
|
||||
screen temporarily.
|
||||
|
||||
### Panels
|
||||
###Panels###
|
||||
|
||||
Chrome recently released a new concept for opening new windows
|
||||
called “Panels”, and once I discovered them I couldn’t get enough of them. The
|
||||
@ -188,7 +187,7 @@ Panel](https://chrome.google.com/webstore/detail/improved-google-tasks-pan/kgnap
|
||||
I’m still lacking Facebook Messenger and Google Voice panel view apps, so I
|
||||
might try my hand at creating one myself soon.
|
||||
|
||||
### Web Browsing
|
||||
###Web Browsing###
|
||||
|
||||
And, of course, being a laptop dedicated to chrome, it
|
||||
obviously has a great web browsing experience.
|
||||
|
||||
@ -13,7 +13,6 @@ features, one of its best being a version control system that allows you to
|
||||
send a draft to other people and accept or reject any changes they suggest. It
|
||||
also has a minamilistic iA Writer type interface, which focuses on the actual
|
||||
writing and nothing more.
|
||||
<!--excerpt-->
|
||||
|
||||
One of my most favorite features that I have just discovered, though, is that
|
||||
it allows publishing any Draft document to any arbitrary
|
||||
|
||||
@ -9,7 +9,6 @@ development knowledge had exceeded what it was showing off. The main thing that
|
||||
annoyed me about my last website was that I was hosting what essentially was a
|
||||
static website on a web framework meant for dynamic websites. It was time for a
|
||||
update.
|
||||
<!--excerpt-->
|
||||
|
||||
I decided to go with [Jekyll](http://jekyllrb.com/) which had everything I
|
||||
wanted:
|
||||
@ -48,5 +47,5 @@ under 20% (on my machine), which was actually better than a few chrome
|
||||
extensions I was running anyways.
|
||||
|
||||
Hopefully this new blog will also inspire me to write more posts as [my last
|
||||
post](/2013/10/03/publishing-draft-docs-to-my-blog.html)
|
||||
post](http://thallada.github.io/2013/10/03/publishing-draft-docs-to-my-blog.html)
|
||||
was almost a year ago now.
|
||||
|
||||
@ -1,268 +0,0 @@
|
||||
---
|
||||
title: Midnight Desktop
|
||||
layout: post
|
||||
---
|
||||
|
||||
I tend to use Linux (Ubuntu) on my desktop late at night in a dark room. To
|
||||
protect my eyes from the blinding light of my monitors I've tooled my desktop
|
||||
environment over the course of a few months to be as dark as possible. It has
|
||||
gotten complex enough that I thought it would be worth sharing now.
|
||||
<!--excerpt-->
|
||||
|
||||
### dotfiles
|
||||
|
||||
Before I begin, I want to note that all the configuration for the setup I'm
|
||||
describing is stored in a [dotfiles repo on my github
|
||||
profile](https://github.com/thallada/dotfiles). If you would like to replicate
|
||||
any of this setup, I would go there. Just note that I will probably be updating
|
||||
the master branch fairly often, but the
|
||||
[midnight](https://github.com/thallada/dotfiles/tree/midnight) branch will
|
||||
always contain the setup described here.
|
||||
|
||||
### bspwm
|
||||
|
||||
Inspired by [/r/unixporn](http://www.reddit.com/r/unixporn), I decided to switch
|
||||
from gnome to bspwm, a minimal tiling window manager that positions windows like
|
||||
leaves on a binary tree.
|
||||
|
||||
I don't really use the tiling features that often, though. I often do most of my
|
||||
work in the terminal and [tmux](http://tmux.sourceforge.net/) does the terminal
|
||||
pane management. But, when I do open another application, it's nice that bspwm
|
||||
forces it to use the maximum available space.
|
||||
|
||||
I also like how hackable the whole manager is. There is a terminal command
|
||||
`bspc` that controls the entire desktop environment and a separate program
|
||||
`sxhkd` (probably the hardest program name ever to remember) handles all of the
|
||||
hotkeys for the environment. All of them are stored in a
|
||||
[`sxhkdrc`](https://github.com/thallada/dotfiles/blob/master/sxhkd/.config/sxhkd/sxhkdrc)
|
||||
under the home directory and it's super easy to add my own. The hotkeys make
|
||||
this superior to gnome for me because I never have to touch my mouse to move
|
||||
around the desktop.
|
||||
|
||||
### gnome and gtk
|
||||
|
||||
I still love some of the features from gnome. Especially the text hinting, which
|
||||
is why I still run `gnome-settings-daemon` in my [bspwm startup
|
||||
script](https://github.com/thallada/dotfiles/blob/master/bspwm/bin/bspwm-session).
|
||||
|
||||
To make gtk applications universally dark (and also to tune text hinting)
|
||||
install the `gnome-tweak-tool`. There should be a "Global Dark Theme" setting
|
||||
under the "Appearance" tab that can be enabled. I use the
|
||||
[Numix](https://numixproject.org/) gtk theme which seems to behave fine with
|
||||
this setting.
|
||||
|
||||
### Gnome Terminal
|
||||
|
||||
I've tried using a few other lighter-weight terminals like xterm, but I still
|
||||
like the features of gnome-terminal more. I created a "bspwm" profile and set
|
||||
the background to be transparent with opacity at about half-way on the slider.
|
||||
My background, set in the [bspwm startup
|
||||
script](https://github.com/thallada/dotfiles/blob/master/bspwm/bin/bspwm-session)
|
||||
is a subtle [dark tiling pattern](https://www.toptal.com/designers/subtlepatterns/mosaic/)
|
||||
so this effectively makes the background of the terminal dark.
|
||||
|
||||
In my
|
||||
[sxhkdrc](https://github.com/thallada/dotfiles/blob/master/sxhkd/.config/sxhkd/sxhkdrc)
|
||||
I can then map my hotkeys for starting a new terminal to the command
|
||||
`gnome-terminal --window-with-profile=bspwm`.
|
||||
|
||||
### vim
|
||||
|
||||
Making vim dark is pretty easy. Just put this in the
|
||||
[`.vimrc`](https://github.com/thallada/dotfiles/blob/master/vim/.vimrc):
|
||||
|
||||
~~~ vim
|
||||
set background=dark
|
||||
~~~
|
||||
|
||||
I use the colorscheme
|
||||
[distinguished](https://github.com/Lokaltog/vim-distinguished) which is
|
||||
installed by putting the `distinguished.vim` file under
|
||||
[`~/.vim/colors/`](https://github.com/thallada/dotfiles/tree/master/vim/.vim/colors)
|
||||
and adding this to the `.vimrc`:
|
||||
|
||||
~~~ vim
|
||||
colorscheme distinguished
|
||||
~~~
|
||||
|
||||
### tmux/byobu
|
||||
|
||||
I like the abstraction that [byobu](http://byobu.co/) puts ontop of tmux, so
|
||||
that's what I use in the terminal. Colors can be configured by editing the
|
||||
[`~/.byobu/color.tmux`](https://github.com/thallada/dotfiles/blob/master/byobu/.byobu/color.tmux)
|
||||
file. This is what I have in mine:
|
||||
|
||||
BYOBU_DARK="\#333333"
|
||||
BYOBU_LIGHT="\#EEEEEE"
|
||||
BYOBU_ACCENT="\#4D2100"
|
||||
BYOBU_HIGHLIGHT="\#303030"
|
||||
MONOCHROME=0
|
||||
|
||||
### evince
|
||||
|
||||
I tell my browser, firefox, to open pdfs in evince (aka. Document Viewer)
|
||||
because evince can darken pdfs.
|
||||
|
||||
Select View > Invert Colors and then Edit > Save Current Settings as Default and
|
||||
now most pdfs will be displayed as white text on black background.
|
||||
|
||||
### gimp
|
||||
|
||||
Gimp allows you to change themes easily. [Gimp GTK2 Photoshop CS6
|
||||
Theme](http://gnome-look.org/content/show.php?content=160952) is my favorite
|
||||
dark theme. Put that in `~/.gimp-2.8/themes/` (or whichever gimp version is
|
||||
installed) and, in Gimp, change the theme at Edit > Preferences > Theme.
|
||||
|
||||
### Firefox
|
||||
|
||||
I had to hack firefox a lot to get it to be universally dark since the web
|
||||
(unfortunately!) doesn't have a night mode switch. I'm using firefox instead of
|
||||
chrome because firefox has better customization for doing something this
|
||||
extreme.
|
||||
|
||||
#### Userstyles
|
||||
|
||||
Firefox has a really neat addon called
|
||||
[Stylish](https://addons.mozilla.org/en-us/firefox/addon/stylish/) that allows
|
||||
you to install and edit user CSS files to change the style of any website you
|
||||
visit. A lot of popular websites have dark themes on
|
||||
[userstyles.org](https://userstyles.org/), but the rest of the internet still
|
||||
mostly has a white background by default.
|
||||
|
||||
Luckily there's a few global dark themes. [Midnight Surfing
|
||||
Alternative](https://userstyles.org/styles/47391/midnight-surfing-alternative)
|
||||
seemed to work the best for me.
|
||||
|
||||
However, since the theme is global, it overwrites the custom tailored dark
|
||||
themes that I had installed for specific popular sites (listed below) making the
|
||||
sites ugly. The Midnight Surfing Alternative theme can be edited through the
|
||||
Stylish extension to exclude the websites that I already have dark themes for.
|
||||
[This superuser question explains what to
|
||||
edit](http://superuser.com/questions/463153/disable-stylish-on-certain-sites-in-firefox).
|
||||
Now, whenever I add a new dark theme to Stylish, I edit the regex to add the
|
||||
domains it covers to the parenthesized list that is delimited by pipes.
|
||||
|
||||
~~~ css
|
||||
@-moz-document regexp("(https?|liberator|file)://(?!([^.]+\\.)?(maps\\.google\\.com|...other domains....)[/:]).*"){
|
||||
~~~
|
||||
|
||||
Here is the list of dark themes I'm currently using with Stylish in addition to
|
||||
Midnight Surfing Alternative:
|
||||
|
||||
* [Amazon Dark -
|
||||
VisualPlastik](https://userstyles.org/styles/52294/amazon-dark-visualplastik)
|
||||
* [Dark Feedly
|
||||
(Hauschild's)](https://userstyles.org/styles/89622/dark-feedly-hauschild-s)
|
||||
* [Dark Gmail mod by
|
||||
Karsonito](https://userstyles.org/styles/107544/dark-gmail-mod-by-karsonito)
|
||||
(this one is a bit buggy right now, though)
|
||||
* [Dark Netflix
|
||||
[GRiMiNTENT]](https://userstyles.org/styles/102627/dark-netflix-grimintent)
|
||||
* [dark-facebook 2 [a dark facebook
|
||||
theme]](https://userstyles.org/styles/95359/facebook-dark-facebook-2-a-dark-facebook-theme)
|
||||
* [Forecast.io - hide
|
||||
map](https://userstyles.org/styles/104812/forecast-io-hide-map)
|
||||
* [GitHub Dark](https://userstyles.org/styles/37035/github-dark) (this one is
|
||||
really well done, I love it)
|
||||
* [Google Play (Music) Dark \*Updated
|
||||
5-15\*](https://userstyles.org/styles/107643/google-play-music-dark-updated-5-15)
|
||||
* [Messenger.com Dark](https://userstyles.org/styles/112722/messenger-com-dark)
|
||||
* [Telegram web dark / custom
|
||||
color](https://userstyles.org/styles/109612/telegram-web-dark-custom-color)
|
||||
* [Youtube - Lights Out - A Dark Youtube
|
||||
Theme](https://userstyles.org/styles/92164/youtube-lights-out-a-dark-youtube-theme)
|
||||
|
||||
#### UI Themes
|
||||
|
||||
Most of my firefox UI is styled dark with the [FT
|
||||
DeepDark](https://addons.mozilla.org/en-US/firefox/addon/ft-deepdark/) theme.
|
||||
|
||||
The firefox developer tools can be [themed dark in its
|
||||
settings](http://soledadpenades.com/2014/11/20/using-the-firefox-developer-edition-dark-theme-with-nightly/).
|
||||
|
||||
#### Addons
|
||||
|
||||
For reddit, I use the [RES](http://redditenhancementsuite.com/) addon which has
|
||||
a night mode option.
|
||||
|
||||
I also use [Custom New
|
||||
Tab](https://addons.mozilla.org/en-US/firefox/addon/custom-new-tab/) combined
|
||||
with [homepage.py](https://github.com/ok100/homepage.py) to display a list of my
|
||||
favorite websites when I start a new tab.
|
||||
|
||||
[Vimperator](https://addons.mozilla.org/en-US/firefox/addon/vimperator/) allows
|
||||
me to control firefox completely with my keyboard which is really useful when I
|
||||
am switching back and forth between firefox and vim. By default, the vimperator
|
||||
window has a white background, so I had to [set it to a dark
|
||||
theme](https://github.com/vimpr/vimperator-colors). Also, in order to make all
|
||||
of the vimperator help pages dark, I had to add the protocol `liberator://` to
|
||||
the regex for Midnight Surfing Alternative (exact syntax for that above).
|
||||
|
||||
### Redshift
|
||||
|
||||
At night, it's also useful to filter out blue light to help with sleep.
|
||||
[Redshift](http://jonls.dk/redshift/) is a utility that does this automatically
|
||||
while running in the background.
|
||||
|
||||
|
||||
|
||||
### Invert it all!
|
||||
|
||||
I noticed that with the dark colors and my monitor brightness turned low, it was
|
||||
hard to see the screen during the day because of glares. An easy solution to
|
||||
this is to simply invert the colors on the output of the monitor into an instant
|
||||
day theme.
|
||||
|
||||
I would have used the command `xcalib -a -i` but that would only work on one
|
||||
monitor and I have two. Luckily, someone made a utility that would invert colors
|
||||
on more than one monitor called
|
||||
[xrandr-invert-colors](https://github.com/zoltanp/xrandr-invert-colors).
|
||||
|
||||
The only problem was that this utility seemed to interfere with redshift, so I
|
||||
made [a script that would disable redshift before
|
||||
inverting](https://github.com/thallada/dotfiles/blob/master/invert/bin/invert).
|
||||
|
||||
~~~ bash
|
||||
#!/bin/bash
|
||||
inverted=$(xcalib -a -p | head -c 1)
|
||||
if [ "$inverted" == "W" ]
|
||||
then
|
||||
if [ -z "$(pgrep redshift)" ]
|
||||
then
|
||||
xrandr-invert-colors
|
||||
redshift &
|
||||
fi
|
||||
else
|
||||
if [ -z "$(pgrep redshift)" ]
|
||||
then
|
||||
xrandr-invert-colors
|
||||
else
|
||||
killall redshift
|
||||
sleep 3
|
||||
xrandr-invert-colors
|
||||
fi
|
||||
fi
|
||||
~~~
|
||||
|
||||
|
||||
And, now I have [a
|
||||
shortcut](https://github.com/thallada/dotfiles/commit/e5153a90fa7c89a0e2ca16e5943f0fa20d4a9512)
|
||||
to invert the screen.
|
||||
|
||||
However, images and videos look pretty ugly inverted. VLC has a setting under
|
||||
Tools > Effects and Filters > Video Effects > Colors called Negate colors that
|
||||
can fix that.
|
||||
|
||||
For firefox, I made a global userstyle to invert images and videos.
|
||||
|
||||
~~~ css
|
||||
@-moz-document regexp("(https?|liberator|file)://(?!([^.]+\\.)?[/:]).*"){
|
||||
img, video, div.html5-video-container, div.player-api, span.emoji, i.emoji, span.emoticon, object[type="application/x-shockwave-flash"], embed[type="application/x-shockwave-flash"] {
|
||||
filter: invert(100%);
|
||||
}
|
||||
}
|
||||
~~~
|
||||
|
||||
Whenever I invert the colors, I enable that global theme on firefox.
|
||||
|
||||
|
||||
@ -1,276 +0,0 @@
|
||||
---
|
||||
title: Generating Realistic Satellite Imagery with Deep Neural Networks
|
||||
layout: post
|
||||
---
|
||||
|
||||
I've been doing a lot of experimenting with [neural-style](https://github.com/jcjohnson/neural-style)
|
||||
the last month. I think I've discovered a few exciting applications of the
|
||||
technique that I haven't seen anyone else do yet. The true power of this
|
||||
algorithm really shines when you can see concrete examples.
|
||||
<!--excerpt-->
|
||||
|
||||
Skip to the **Applications** part of this post to see the outputs from my
|
||||
experimentation if you are already familiar with DeepDream, Deep Style, and all
|
||||
the other latest happenings in generating images with deep neural networks.
|
||||
|
||||
### Background and History
|
||||
|
||||
On [May 18, 2015 at 2 a.m., Alexander
|
||||
Mordvintsev](https://medium.com/backchannel/inside-deep-dreams-how-google-made-its-computers-go-crazy-83b9d24e66df#.g4t69y8wy),
|
||||
an engineer at Google, did something with deep neural networks that no one had
|
||||
done before. He took a net designed for *recognizing* objects in images and used
|
||||
it to *generate* objects in images. In a sense, he was telling these systems
|
||||
that mimic the human visual cortex to hallucinate things that weren't really
|
||||
there. The [results](https://i.imgur.com/6ocuQsZ.jpg) looked remarkably like LSD
|
||||
trips or what a [schizophrenic person sees on a blank
|
||||
wall](https://www.reddit.com/r/deepdream/comments/3cewgn/an_artist_suffering_from_schizophrenia_was_told/).
|
||||
|
||||
Mordvintsev's discovery quickly gathered attention at Google once he posted
|
||||
images from his experimentation on the company's internal network. On June 17,
|
||||
2015, [Google posted a blog post about the
|
||||
technique](http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html)
|
||||
(dubbed "Inceptionism") and how it was useful for opening up the notoriously
|
||||
black-boxed neural networks using visualizations that researchers could examine.
|
||||
These machine hallucinations were key for identifying the features of objects
|
||||
that neural networks used to tell one object from another (like a dog from a
|
||||
cat). But the post also revealed the [beautiful
|
||||
results](https://goo.gl/photos/fFcivHZ2CDhqCkZdA) of applying the algorithm
|
||||
iteratively on it's own outputs and zooming out at each step.
|
||||
|
||||
The internet exploded in response to this post. And once [Google posted the code
|
||||
for performing the
|
||||
technique](http://googleresearch.blogspot.com/2015/07/deepdream-code-example-for-visualizing.html?m=1),
|
||||
people began experimenting and sharing [their fantastic and creepy
|
||||
images](https://www.reddit.com/r/deepdream) with the world.
|
||||
|
||||
Then, on August, 26, 2015, a paper titled ["A Neural Algorithm of Artistic
|
||||
Style"](http://arxiv.org/abs/1508.06576) was published. It showed how one could
|
||||
identify which layers of deep neural networks recognized stylistic information
|
||||
of an image (and not the content) and then use this stylistic information in
|
||||
Google's Inceptionism technique to paint other images in the style of any
|
||||
artist. A [few](https://github.com/jcjohnson/neural-style)
|
||||
[implementations](https://github.com/kaishengtai/neuralart) of the paper were
|
||||
put up on Github. This exploded the internet again in a frenzy. This time, the
|
||||
images produced were less like psychedelic-induced nightmares but more like the
|
||||
next generation of Instagram filters ([reddit
|
||||
how-to](https://www.reddit.com/r/deepdream/comments/3jwl76/how_anyone_can_create_deep_style_images/)).
|
||||
|
||||
People began to wonder [what all of this
|
||||
meant](http://www.hopesandfears.com/hopes/culture/is-this-art/215039-deep-dream-google-art)
|
||||
to [the future of
|
||||
art](http://kajsotala.fi/2015/07/deepdream-today-psychedelic-images-tomorrow-unemployed-artists/).
|
||||
Some of the results produced where [indistinguishable from the style of dead
|
||||
artists'
|
||||
works](https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/outputs/tubingen_starry.png).
|
||||
Was this a demonstration of creativity in computers or just a neat trick?
|
||||
|
||||
On November, 19, 2015, [another paper](http://arxiv.org/abs/1511.06434) was
|
||||
released that demonstrated a technique for generating scenes from convolutional
|
||||
neural nets ([implementation on Github](https://github.com/Newmu/dcgan_code)).
|
||||
The program could generate random (and very realistic) [bedroom
|
||||
images](https://github.com/Newmu/dcgan_code/raw/master/images/lsun_bedrooms_five_epoch_samples.png)
|
||||
from a neural net trained on bedroom images. Amazingly, it could also generate
|
||||
[the same bedroom from any
|
||||
angle](https://github.com/Newmu/dcgan_code/blob/master/images/lsun_bedrooms_five_epochs_interps.png).
|
||||
It could also [produce images of the same procedurally generated face from any
|
||||
angle](https://github.com/Newmu/dcgan_code/blob/master/images/turn_vector.png).
|
||||
Theoretically, we could use this technology to create *procedurally generated
|
||||
game art*.
|
||||
|
||||
The main thing holding this technology back from revolutionizing procedurally
|
||||
generated video games is that it is not real-time. Using
|
||||
[neural-style](https://github.com/jcjohnson/neural-style) to apply artistic
|
||||
style to a 512 by 512 pixel content image could take minutes even on the
|
||||
top-of-the-line GTX Titan X graphics card. Still, I believe this technology has
|
||||
a lot of potential for generating game art even if it can't act as a real-time
|
||||
filter.
|
||||
|
||||
### Applications: Generating Satellite Images for Procedural World Maps
|
||||
|
||||
I personally know very little machine learning, but I have been able to produce
|
||||
a lot of interesting results by using the tool provided by
|
||||
[neural-style](https://github.com/jcjohnson/neural-style).
|
||||
|
||||
Inspired by [Kaelan's procedurally generated world
|
||||
maps](http://blog.kaelan.org/randomly-generated-world-map/), I wanted to extend
|
||||
the idea by generating realistic satellite images of the terrain maps. The
|
||||
procedure is simple: take a [generated terrain map](/assets/kaelan_terrain1.png)
|
||||
and apply the style of a [real-world satellite image](/assets/uk_satellite.jpg)
|
||||
on it using neural-style.
|
||||
|
||||
|
||||
|
||||
The generated output takes on whatever terrain is in the satellite image. Here
|
||||
is an output processing one of Kaelan's maps with a [arctic satellite
|
||||
image](/assets/svalbard_satellite.jpg):
|
||||
|
||||
|
||||
|
||||
|
||||
And again, with one of Kaelan's desert maps and a [satellite image of a
|
||||
desert](/assets/desert_satellite.jpg):
|
||||
|
||||
|
||||
|
||||
|
||||
It even works with [Kaelan's generated hexagon
|
||||
maps](http://blog.kaelan.org/hexagon-world-map-generation/). Here's an island
|
||||
hexagon map plus a [satellite image of a volcanic
|
||||
island](/assets/volcano_satellite.jpg):
|
||||
|
||||
|
||||
|
||||
|
||||
This image even produced an interesting three-dimensional effect because of the
|
||||
volcano in the satellite image.
|
||||
|
||||
By the way, this also works with minecraft maps. Here's a minecraft map I found
|
||||
on the internet plus a [satellite image from Google
|
||||
Earth](/assets/river_satellite.png):
|
||||
|
||||
|
||||
|
||||
|
||||
No fancy texture packs or 3-D rendering needed :).
|
||||
|
||||
Here is the Fallout 4 grayscale map plus a
|
||||
[satellite image of Boston](/assets/boston_aerial.jpg):
|
||||
|
||||
|
||||
|
||||
|
||||
Unfortunately, it puts the built-up dense part of the city in the wrong part of
|
||||
the geographic area. But, this is understandable since we gave the algorithm no
|
||||
information on where that is on the map.
|
||||
|
||||
We can also make the generated terrain maps look like old hand-drawn maps using
|
||||
neural-style. With Kaelan's terrain map as the
|
||||
content and [the in-game Elder Scrolls IV Oblivion map of
|
||||
Cyrodiil](/assets/cyrodiil_ingame.jpg) as the style we get this:
|
||||
|
||||
|
||||
|
||||
|
||||
It looks cool, but the water isn't conveyed very clearly (e.g. makes deep water
|
||||
look like land). Neural-style seems to work better when there is lots of color
|
||||
in both images.
|
||||
|
||||
Here is the output of the hex terrain plus satellite map above and the Cyrodiil
|
||||
map which looks a little cleaner:
|
||||
|
||||
|
||||
|
||||
|
||||
I was interested to see what neural-style could generate from random noise, so I
|
||||
rendered some clouds in GIMP and ran it with a satellite image of [Mexico City
|
||||
from Google Earth](/assets/mexico_city.jpg) (by the way, I've been getting high
|
||||
quality Google Earth shots from
|
||||
[earthview.withgoogle.com](https://earthview.withgoogle.com)).
|
||||
|
||||
|
||||
|
||||
|
||||
Not bad for a neural net without a degree in urban planning.
|
||||
|
||||
I also tried generating on random noise with a satellite image of [a water
|
||||
treatment plant in Peru](/assets/treatment_plant.jpg)
|
||||
|
||||
|
||||
|
||||
|
||||
### Applications: More Fun
|
||||
|
||||
For fun, here are some other outputs that I liked.
|
||||
|
||||
[My photo of Boston's skyline as the content](/assets/boston_skyline.jpg) and
|
||||
[Vincent van Gogh's The Starry Night as the style](/assets/starry_night.jpg):
|
||||
|
||||
|
||||
|
||||
[A photo of me](/assets/standing_forest.jpg) (by Aidan Bevacqua) and [Forrest in
|
||||
the end of Autumn by Caspar David Friedrich](/assets/forrest_autumn.jpg):
|
||||
|
||||
|
||||
|
||||
[Another photo of me by Aidan](/assets/sitting_forest.jpg) in the same style:
|
||||
|
||||
|
||||
|
||||
[A photo of me on a mountain](/assets/mountain_view.jpg) (by Aidan Bevacqua) and
|
||||
[pixel art by Paul Robertson](/assets/pixels.png)
|
||||
|
||||
|
||||
|
||||
[A photo of a park in Copenhagen I took](/assets/copenhagen_park.jpg) and a
|
||||
painting similar in composition, [Avenue of Poplars at Sunset by Vincent van
|
||||
Gogh](/assets/avenue_poplars.jpg):
|
||||
|
||||
|
||||
|
||||
[My photo of the Shenandoah National Park](/assets/shenandoah_mountains.jpg) and
|
||||
[this halo graphic from GMUNK](/assets/halo_ring_mountains.jpg)
|
||||
([GMUNK](http://www.gmunk.com/filter/Interactive/ORA-Summoners-HALO)):
|
||||
|
||||
|
||||
|
||||
[A photo of me by Aidan](/assets/me.png) and a [stained glass
|
||||
fractal](/assets/stained_glass.jpg):
|
||||
|
||||
|
||||
|
||||
Same photo of me and some [psychedelic art by GMUNK](/assets/pockets.jpg)
|
||||
|
||||
|
||||
|
||||
[New York City](/assets/nyc.jpg) and [a rainforest](/assets/rainforest.jpg):
|
||||
|
||||
|
||||
|
||||
[Kowloon Walled City](/assets/kowloon.jpg) and [a National Geographic
|
||||
Map](/assets/ngs_map.jpg):
|
||||
|
||||
|
||||
|
||||
[A photo of me by Aidan](/assets/side_portrait.jpg) and [Head of Lioness by
|
||||
Theodore Gericault](/assets/head_lioness.jpg):
|
||||
|
||||
|
||||
|
||||
[Photo I took of a Norwegian forest](/assets/forest_hill.jpg) and [The Mountain
|
||||
Brook by Albert Bierstadt](/assets/mountain_brook.jpg):
|
||||
|
||||
|
||||
|
||||
### Limitations
|
||||
|
||||
I don't have infinite money for a GTX Titan X, so I'm stuck with using OpenCL on
|
||||
my more-than-a-few-generations-old AMD card. It takes about a half-hour to
|
||||
generate one 512x512 px image in my set-up (which makes the feedback loop for
|
||||
correcting mistakes *very* long). And sometimes the neural-style refuses to run
|
||||
on my GPU (I suspect it runs out of VRAM), so I have to run it on my CPU which
|
||||
takes even longer...
|
||||
|
||||
I am unable to generate bigger images (though
|
||||
[the author has been able to generate up to 1920x1010
|
||||
px](https://github.com/jcjohnson/neural-style/issues/36#issuecomment-142994812)).
|
||||
As the size of the output increases the amount of memory and time to generate
|
||||
also increases. And, it's not practical to just generate thumbnails to test
|
||||
parameters, because increasing the image size will probably generate a very
|
||||
different image since all the other parameters stay the same even though they
|
||||
are dependent on the image size.
|
||||
|
||||
Some people have had success running these neural nets on GPU spot instances in
|
||||
AWS. It would be certainly cheaper than buying a new GPU in the short-term.
|
||||
|
||||
So, I have a few more ideas for what to run, but it will take me quite a while
|
||||
to get through the queue.
|
||||
@ -1,197 +0,0 @@
|
||||
---
|
||||
title: How to Install TensorFlow on Ubuntu 16.04 with GPU Support
|
||||
layout: post
|
||||
---
|
||||
|
||||
I found the [tensorflow
|
||||
documentation](https://www.tensorflow.org/install/install_linux) rather lacking
|
||||
for installation instructions, especially in regards to getting GPU support.
|
||||
I'm going to write down my notes from wrangling with the installation here for
|
||||
future reference and hopefully this helps someone else too.
|
||||
<!--excerpt-->
|
||||
|
||||
This will invariably go out-of-date at some point, so be mindful of the publish
|
||||
date of this post. Make sure to cross-reference other documentation that has
|
||||
more up-to-date information.
|
||||
|
||||
## Assumptions
|
||||
|
||||
These instructions are very specific to my environment, so this is what I am
|
||||
assuming:
|
||||
|
||||
1. You are running Ubuntu 16.04. (I have 16.04.1)
|
||||
- You can check this in the output of `uname -a`
|
||||
2. You have a 64 bit machine.
|
||||
- You can check this with `uname -m`. (should say `x86_64`)
|
||||
2. You have an NVIDIA GPU that has CUDA Compute Capability 3.0 or higher.
|
||||
[NVIDIA documentation](https://developer.nvidia.com/cuda-gpus) has a full table
|
||||
of cards and their Compute Capabilities. (I have a GeForce GTX 980 Ti)
|
||||
- You can check what card you have in Settings > Details under the label
|
||||
"Graphics"
|
||||
- You can also check by verifying there is any output when you run `lspci |
|
||||
grep -i nvidia`
|
||||
3. You have a linux kernel version 4.4.0 or higher. (I have 4.8.0)
|
||||
- You can check this by running `uname -r`
|
||||
4. You have gcc version 5.3.1 or higher installed. (I have 5.4.0)
|
||||
- You can check this by running `gcc --version`
|
||||
5. You have the latest [proprietary](https://i.imgur.com/8osspXj.jpg) NVIDIA
|
||||
drivers installed.
|
||||
- You can check this and install it if you haven't in the "Additional
|
||||
Drivers" tab in the "Software & Updates" application (`update-manager`).
|
||||
(I have version 375.66 installed)
|
||||
6. You have the kernel headers installed.
|
||||
- Just run `sudo apt-get install linux-headers-$(uname -r)` to install them
|
||||
if you don't have them installed already.
|
||||
7. You have Python installed. The exact version shouldn't matter, but for the
|
||||
rest of this post I'm going to assume you have `python3` installed.
|
||||
- You can install `python3` by running `sudo apt-get install python3`. This
|
||||
will install Python 3.5.
|
||||
- Bonus points: you can install Python 3.6 by following [this
|
||||
answer](https://askubuntu.com/a/865569), but Python 3.5 should be fine.
|
||||
|
||||
## Install the CUDA Toolkit 8.0
|
||||
|
||||
NVIDIA has [a big scary documentation
|
||||
page](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/) on this, but I
|
||||
will summarize the only the parts you need to know here.
|
||||
|
||||
Go to the [CUDA Toolkit Download](https://developer.nvidia.com/cuda-downloads)
|
||||
page. Click Linux > x86_64 > Ubuntu > 16.04 > deb (network).
|
||||
|
||||
Click download and then follow the instructions, copied here:
|
||||
|
||||
1. `sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb`
|
||||
2. `sudo apt-get update`
|
||||
3. `sudo apt-get install cuda`
|
||||
|
||||
This will install CUDA 8.0. It installed it to the directory
|
||||
`/usr/local/cuda-8.0/` on my machine.
|
||||
|
||||
There are some [post-install
|
||||
actions](http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions)
|
||||
we must follow:
|
||||
|
||||
1. Edit your `~/.bashrc`
|
||||
- Use your favorite editor `gedit ~/.bashrc`, `nano ~/.bashrc`, `vim
|
||||
~/.bashrc`, whatever.
|
||||
2. Add the following lines to the end of the file:
|
||||
```bash
|
||||
# CUDA 8.0 (nvidia) paths
|
||||
export CUDA_HOME=/usr/local/cuda-8.0
|
||||
export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
|
||||
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
|
||||
```
|
||||
3. Save and exit.
|
||||
4. Run `source ~/.bashrc`.
|
||||
5. Install writable samples by running the script `cuda-install-samples-8.0.sh
|
||||
~/`.
|
||||
- If the script cannot be found, the above steps didn't work :(
|
||||
- I don't actually know if the samples are absolutely required for what I'm
|
||||
using CUDA for, but it's recommended according to NVIDIA, and compiling
|
||||
them will output a nifty `deviceQuery` binary which can be ran to test if
|
||||
everything is working properly.
|
||||
6. Make sure `nvcc -V` outputs something.
|
||||
- If an error, the above steps 1-4 didn't work :(
|
||||
7. `cd ~/NVIDIA_CUDA-8.0_Samples`, cross your fingers, and run `make`
|
||||
- The compile will take a while
|
||||
- My compile actually errored near the end with an error about `/usr/bin/ld:
|
||||
cannot find -lnvcuvid` I *think* that doesn't really matter because the
|
||||
binary files were still output.
|
||||
8. Try running `~/NVIDIA_CUDA-8.0_Samples/bin/x86_64/linux/release/deviceQuery`
|
||||
to see if you get any output. Hopefully you will see your GPU listed.
|
||||
|
||||
## Install cuDNN v5.1
|
||||
|
||||
[This AskUbuntu answer](https://askubuntu.com/a/767270) has good instructions.
|
||||
Here are the instructions specific to this set-up:
|
||||
|
||||
1. Visit the [NVIDIA cuDNN page](https://developer.nvidia.com/cudnn) and click
|
||||
"Download".
|
||||
2. Join the program and fill out the survey.
|
||||
3. Agree to the terms of service.
|
||||
4. Click the link for "Download cuDNN v5.1 (Jan 20, 2017), for CUDA 8.0"
|
||||
5. Download the "cuDNN v5.1 Library for Linux" (3rd link from the top).
|
||||
6. Untar the downloaded file. E.g.:
|
||||
```bash
|
||||
cd ~/Downloads
|
||||
tar -xvf cudnn-8.0-linux-x64-v5.1.tgz
|
||||
```
|
||||
7. Install the cuDNN files to the CUDA folder:
|
||||
```bash
|
||||
cd cuda
|
||||
sudo cp -P include/* /usr/local/cuda-8.0/include/
|
||||
sudo cp -P lib64/* /usr/local/cuda-8.0/lib64/
|
||||
sudo chmod a+r /usr/local/cuda-8.0/lib64/libcudnn*
|
||||
```
|
||||
|
||||
## Install libcupti-dev
|
||||
|
||||
This one is simple. Just run:
|
||||
|
||||
```bash
|
||||
sudo apt-get install libcupti-dev
|
||||
```
|
||||
|
||||
## Create a Virtualenv
|
||||
|
||||
I recommend using
|
||||
[virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/index.html)
|
||||
to create the tensorflow virtualenv, but the TensorFlow docs still have
|
||||
[instructions to create the virtualenv
|
||||
manually](https://www.tensorflow.org/install/install_linux#InstallingVirtualenv).
|
||||
|
||||
1. [Install
|
||||
virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/install.html).
|
||||
Make sure to add [the required
|
||||
lines](https://virtualenvwrapper.readthedocs.io/en/latest/install.html#shell-startup-file)
|
||||
to your `~/.bashrc`.
|
||||
2. Create the virtualenv:
|
||||
```bash
|
||||
mkvirtualenv --python=python3 tensorflow
|
||||
```
|
||||
|
||||
## Install the TensorFlow with GPU support
|
||||
|
||||
If you just run `pip install tensorflow` you will not get GPU support. To
|
||||
install the correct version you will have to install from a [particular
|
||||
url](https://www.tensorflow.org/install/install_linux#python_35). Here is the
|
||||
install command you will have to run to install TensorFlow 1.2 for Python 3.5
|
||||
with GPU support:
|
||||
|
||||
```bash
|
||||
pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.2.0-cp35-cp35m-linux_x86_64.whl
|
||||
```
|
||||
|
||||
If you need a different version of TensorFlow, you can edit the version number
|
||||
in the URL. Same with the Python version (change `cp35` to `cp36` to install for
|
||||
Python 3.6 instead, for example).
|
||||
|
||||
## Test that the installation worked
|
||||
|
||||
Save this script from [the TensorFlow
|
||||
tutorials](https://www.tensorflow.org/tutorials/using_gpu#logging_device_placement)
|
||||
to a file called `test_gpu.py`:
|
||||
|
||||
```python
|
||||
# Creates a graph.
|
||||
with tf.device('/cpu:0'):
|
||||
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
|
||||
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
|
||||
c = tf.matmul(a, b)
|
||||
# Creates a session with log_device_placement set to True.
|
||||
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
|
||||
# Runs the op.
|
||||
print(sess.run(c))
|
||||
```
|
||||
|
||||
And then run it:
|
||||
|
||||
```bash
|
||||
python test_gpu.py
|
||||
```
|
||||
|
||||
You should see your GPU card listed under "Device mapping:" and that each task
|
||||
in the compute graph is assigned to `gpu:0`.
|
||||
|
||||
If you see "Device mapping: no known devices" then something went wrong and
|
||||
TensorFlow cannot access your GPU.
|
||||
@ -1,516 +0,0 @@
|
||||
---
|
||||
title: Generating Random Poems with Python
|
||||
layout: post
|
||||
image: /img/blog/buzzfeed.jpg
|
||||
---
|
||||
|
||||
In this post, I will demonstrate how to generate random text using a few lines
|
||||
of standard python and then progressively refine the output until it looks
|
||||
poem-like.
|
||||
|
||||
If you would like to follow along with this post and run the code snippets
|
||||
yourself, you can clone [my NLP repository](https://github.com/thallada/nlp/)
|
||||
and run [the Jupyter
|
||||
notebook](https://github.com/thallada/nlp/blob/master/edX%20Lightning%20Talk.ipynb).
|
||||
|
||||
You might not realize it, but you probably use an app everyday that can generate
|
||||
random text that sounds like you: your phone keyboard.
|
||||
<!--excerpt-->
|
||||
|
||||
|
||||
|
||||
Just by tapping the next suggested word over and over, you can generate text. So how does it work?
|
||||
|
||||
## Corpus
|
||||
|
||||
First, we need a **corpus**: the text our generator will recombine into new
|
||||
sentences. In the case of your phone keyboard, this is all the text you've ever
|
||||
typed into your keyboard. For our example, let's just start with one sentence:
|
||||
|
||||
```python
|
||||
corpus = 'The quick brown fox jumps over the lazy dog'
|
||||
```
|
||||
|
||||
## Tokenization
|
||||
|
||||
Now we need to split this corpus into individual **tokens** that we can operate
|
||||
on. Since our objective is to eventually predict the next word from the previous
|
||||
word, we will want our tokens to be individual words. This process is called
|
||||
**tokenization**. The simplest way to tokenize a sentence into words is to split
|
||||
on spaces:
|
||||
|
||||
```python
|
||||
words = corpus.split(' ')
|
||||
words
|
||||
```
|
||||
```python
|
||||
['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
|
||||
```
|
||||
|
||||
## Bigrams
|
||||
|
||||
Now, we will want to create **bigrams**. A bigram is a pair of two words that
|
||||
are in the order they appear in the corpus. To create bigrams, we will iterate
|
||||
through the list of the words with two indices, one of which is offset by one:
|
||||
|
||||
```python
|
||||
bigrams = [b for b in zip(words[:-1], words[1:])]
|
||||
bigrams
|
||||
```
|
||||
```python
|
||||
[('The', 'quick'),
|
||||
('quick', 'brown'),
|
||||
('brown', 'fox'),
|
||||
('fox', 'jumps'),
|
||||
('jumps', 'over'),
|
||||
('over', 'the'),
|
||||
('the', 'lazy'),
|
||||
('lazy', 'dog')]
|
||||
```
|
||||
|
||||
How do we use the bigrams to predict the next word given the first word?
|
||||
|
||||
Return every second element where the first element matches the **condition**:
|
||||
|
||||
```python
|
||||
condition = 'the'
|
||||
next_words = [bigram[1] for bigram in bigrams
|
||||
if bigram[0].lower() == condition]
|
||||
next_words
|
||||
```
|
||||
```python
|
||||
['quick', 'lazy']
|
||||
```
|
||||
|
||||
We have now found all of the possible words that can follow the condition "the"
|
||||
according to our corpus: "quick" and "lazy".
|
||||
|
||||
<pre>
|
||||
(<span style="color:blue">The</span> <span style="color:red">quick</span>) (quick brown) ... (<span style="color:blue">the</span> <span style="color:red">lazy</span>) (lazy dog)
|
||||
</pre>
|
||||
|
||||
Either "<span style="color:red">quick</span>" or "<span
|
||||
style="color:red">lazy</span>" could be the next word.
|
||||
|
||||
## Trigrams and N-grams
|
||||
|
||||
We can partition our corpus into groups of threes too:
|
||||
|
||||
<pre>
|
||||
(<span style="color:blue">The</span> <span style="color:red">quick brown</span>) (quick brown fox) ... (<span style="color:blue">the</span> <span style="color:red">lazy dog</span>)
|
||||
</pre>
|
||||
|
||||
Or, the condition can be two words (`condition = 'the lazy'`):
|
||||
|
||||
<pre>
|
||||
(The quick brown) (quick brown fox) ... (<span style="color:blue">the lazy</span> <span style="color:red">dog</span>)
|
||||
</pre>
|
||||
|
||||
These are called **trigrams**.
|
||||
|
||||
We can partition any **N** number of words together as **n-grams**.
|
||||
|
||||
## Conditional Frequency Distributions
|
||||
|
||||
Earlier, we were able to compute the list of possible words to follow a
|
||||
condition:
|
||||
|
||||
```python
|
||||
next_words
|
||||
```
|
||||
```python
|
||||
['quick', 'lazy']
|
||||
```
|
||||
|
||||
But, in order to predict the next word, what we really want to compute is what
|
||||
is the most likely next word out of all of the possible next words. In other
|
||||
words, find the word that occurred the most often after the condition in the
|
||||
corpus.
|
||||
|
||||
We can use a **Conditional Frequency Distribution (CFD)** to figure that out! A
|
||||
**CFD** can tell us: given a **condition**, what is **likelihood** of each
|
||||
possible outcome.
|
||||
|
||||
This is an example of a CFD with two conditions, displayed in table form. It is
|
||||
counting words appearing in a text collection (source: nltk.org).
|
||||
|
||||
|
||||
|
||||
Let's change up our corpus a little to better demonstrate the CFD:
|
||||
|
||||
```python
|
||||
words = ('The quick brown fox jumped over the '
|
||||
'lazy dog and the quick cat').split(' ')
|
||||
print words
|
||||
```
|
||||
```python
|
||||
['The', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy', 'dog', 'and', 'the', 'quick', 'cat']
|
||||
```
|
||||
|
||||
Now, let's build the CFD. I use
|
||||
[`defaultdicts`](https://docs.python.org/2/library/collections.html#defaultdict-objects)
|
||||
to avoid having to initialize every new dict.
|
||||
|
||||
```python
|
||||
from collections import defaultdict
|
||||
cfd = defaultdict(lambda: defaultdict(lambda: 0))
|
||||
for i in range(len(words) - 2): # loop to the next-to-last word
|
||||
cfd[words[i].lower()][words[i+1].lower()] += 1
|
||||
|
||||
# pretty print the defaultdict
|
||||
{k: dict(v) for k, v in dict(cfd).items()}
|
||||
```
|
||||
```python
|
||||
{'and': {'the': 1},
|
||||
'brown': {'fox': 1},
|
||||
'dog': {'and': 1},
|
||||
'fox': {'jumped': 1},
|
||||
'jumped': {'over': 1},
|
||||
'lazy': {'dog': 1},
|
||||
'over': {'the': 1},
|
||||
'quick': {'brown': 1},
|
||||
'the': {'lazy': 1, 'quick': 2}}
|
||||
```
|
||||
|
||||
So, what's the most likely word to follow `'the'`?
|
||||
|
||||
```python
|
||||
max(cfd['the'])
|
||||
```
|
||||
```python
|
||||
'quick'
|
||||
```
|
||||
|
||||
Whole sentences can be the conditions and values too. Which is basically the way
|
||||
[cleverbot](http://www.cleverbot.com/) works.
|
||||
|
||||
|
||||
|
||||
## Random Text
|
||||
|
||||
Lets put this all together, and with a little help from
|
||||
[nltk](http://www.nltk.org/) generate some random text.
|
||||
|
||||
```python
|
||||
import nltk
|
||||
import random
|
||||
|
||||
TEXT = nltk.corpus.gutenberg.words('austen-emma.txt')
|
||||
|
||||
# NLTK shortcuts :)
|
||||
bigrams = nltk.bigrams(TEXT)
|
||||
cfd = nltk.ConditionalFreqDist(bigrams)
|
||||
|
||||
# pick a random word from the corpus to start with
|
||||
word = random.choice(TEXT)
|
||||
# generate 15 more words
|
||||
for i in range(15):
|
||||
print word,
|
||||
if word in cfd:
|
||||
word = random.choice(cfd[word].keys())
|
||||
else:
|
||||
break
|
||||
```
|
||||
|
||||
Which outputs something like:
|
||||
|
||||
```
|
||||
her reserve and concealment towards some feelings in moving slowly together .
|
||||
You will shew
|
||||
```
|
||||
|
||||
Great! This is basically what the phone keyboard suggestions are doing. Now how
|
||||
do we take this to the next level and generate text that looks like a poem?
|
||||
|
||||
## Random Poems
|
||||
|
||||
Generating random poems is accomplished by limiting the choice of the next word
|
||||
by some constraint:
|
||||
|
||||
* words that rhyme with the previous line
|
||||
* words that match a certain syllable count
|
||||
* words that alliterate with words on the same line
|
||||
* etc.
|
||||
|
||||
## Rhyming
|
||||
|
||||
### Written English != Spoken English
|
||||
|
||||
English has a highly **nonphonemic orthography**, meaning that the letters often
|
||||
have no correspondence to the pronunciation. E.g.:
|
||||
|
||||
> "meet" vs. "meat"
|
||||
|
||||
The vowels are spelled differently, yet they rhyme [^1].
|
||||
|
||||
So if the spelling of the words is useless in telling us if two words rhyme,
|
||||
what can we use instead?
|
||||
|
||||
### International Phonetic Alphabet (IPA)
|
||||
|
||||
The IPA is an alphabet that can represent all varieties of human pronunciation.
|
||||
|
||||
* meet: /mit/
|
||||
* meat: /mit/
|
||||
|
||||
Note that this is only the IPA transcription for only one **accent** of English.
|
||||
Some English speakers may pronounce these words differently which could be
|
||||
represented by a different IPA transcription.
|
||||
|
||||
## Syllables
|
||||
|
||||
How can we determine the number of syllables in a word? Let's consider the two
|
||||
words "poet" and "does":
|
||||
|
||||
* "poet" = 2 syllables
|
||||
* "does" = 1 syllable
|
||||
|
||||
The vowels in these two words are written the same, but are pronounced
|
||||
differently with a different number of syllables.
|
||||
|
||||
Can the IPA tell us the number of syllables in a word too?
|
||||
|
||||
* poet: /ˈpoʊət/
|
||||
* does: /ˈdʌz/
|
||||
|
||||
Not really... We cannot easily identify the number of syllables from those
|
||||
transcriptions. Sometimes the transcriber denotes syllable breaks with a `.` or
|
||||
a `'`, but sometimes they don't.
|
||||
|
||||
### Arpabet
|
||||
|
||||
The Arpabet is a phonetic alphabet developed by ARPA in the 70s that:
|
||||
|
||||
* Encodes phonemes specific to American English.
|
||||
* Meant to be a machine readable code. It is ASCII only.
|
||||
* Denotes how stressed every vowel is from 0-2.
|
||||
|
||||
This is perfect! Because of that third bullet, a word's syllable count equals
|
||||
the number of digits in the Arpabet encoding.
|
||||
|
||||
### CMU Pronouncing Dictionary (CMUdict)
|
||||
|
||||
A large open source dictionary of English words to North American pronunciations
|
||||
in Arpanet encoding. Conveniently, it is also in NLTK...
|
||||
|
||||
### Counting Syllables
|
||||
|
||||
```python
|
||||
import string
|
||||
from nltk.corpus import cmudict
|
||||
cmu = cmudict.dict()
|
||||
|
||||
def count_syllables(word):
|
||||
lower_word = word.lower()
|
||||
if lower_word in cmu:
|
||||
return max([len([y for y in x if y[-1] in string.digits])
|
||||
for x in cmu[lower_word]])
|
||||
|
||||
print("poet: {}\ndoes: {}".format(count_syllables("poet"),
|
||||
count_syllables("does")))
|
||||
```
|
||||
|
||||
Results in:
|
||||
|
||||
```
|
||||
poet: 2
|
||||
does: 1
|
||||
```
|
||||
|
||||
## Buzzfeed Haiku Generator
|
||||
|
||||
To see this in action, try out a haiku generator I created that uses Buzzfeed
|
||||
article titles as a corpus. It does not incorporate rhyming, it just counts the
|
||||
syllables to make sure it's [5-7-5](https://en.wikipedia.org/wiki/Haiku). You can view the full code
|
||||
[here](https://github.com/thallada/nlp/blob/master/generate_poem.py).
|
||||
|
||||
|
||||
|
||||
Run it live at:
|
||||
[http://mule.hallada.net/nlp/buzzfeed-haiku-generator/](http://mule.hallada.net/nlp/buzzfeed-haiku-generator/)
|
||||
|
||||
## Syntax-aware Generation
|
||||
|
||||
Remember these?
|
||||
|
||||
|
||||
|
||||
Mad Libs worked so well because they forced the random words (chosen by the
|
||||
players) to fit into the syntactical structure and parts-of-speech of an
|
||||
existing sentence.
|
||||
|
||||
You end up with **syntactically** correct sentences that are **semantically**
|
||||
random. We can do the same thing!
|
||||
|
||||
### NLTK Syntax Trees!
|
||||
|
||||
NLTK can parse any sentence into a [syntax
|
||||
tree](http://www.nltk.org/book/ch08.html). We can utilize this syntax tree
|
||||
during poetry generation.
|
||||
|
||||
```python
|
||||
from stat_parser import Parser
|
||||
parsed = Parser().parse('The quick brown fox jumps over the lazy dog.')
|
||||
print parsed
|
||||
```
|
||||
|
||||
Syntax tree output as an
|
||||
[s-expression](https://en.wikipedia.org/wiki/S-expression):
|
||||
|
||||
```
|
||||
(S
|
||||
(NP (DT the) (NN quick))
|
||||
(VP
|
||||
(VB brown)
|
||||
(NP
|
||||
(NP (JJ fox) (NN jumps))
|
||||
(PP (IN over) (NP (DT the) (JJ lazy) (NN dog)))))
|
||||
(. .))
|
||||
```
|
||||
|
||||
```python
|
||||
parsed.pretty_print()
|
||||
```
|
||||
|
||||
And the same tree visually pretty printed in ASCII:
|
||||
|
||||
```
|
||||
S
|
||||
________________________|__________________________
|
||||
| VP |
|
||||
| ____|_____________ |
|
||||
| | NP |
|
||||
| | _________|________ |
|
||||
| | | PP |
|
||||
| | | ________|___ |
|
||||
NP | NP | NP |
|
||||
___|____ | ___|____ | _______|____ |
|
||||
DT NN VB JJ NN IN DT JJ NN .
|
||||
| | | | | | | | | |
|
||||
the quick brown fox jumps over the lazy dog .
|
||||
```
|
||||
|
||||
NLTK also performs [part-of-speech tagging](http://www.nltk.org/book/ch05.html)
|
||||
on the input sentence and outputs the tag at each node in the tree. Here's what
|
||||
each of those mean:
|
||||
|
||||
|**S** | Sentence |
|
||||
|**VP** | Verb Phrase |
|
||||
|**NP** | Noun Phrase |
|
||||
|**DT** | Determiner |
|
||||
|**NN** | Noun (common, singular) |
|
||||
|**VB** | Verb (base form) |
|
||||
|**JJ** | Adjective (or numeral, ordinal) |
|
||||
|**.** | Punctuation |
|
||||
|
||||
Now, let's use this information to swap matching syntax sub-trees between two
|
||||
corpora ([source for the generate
|
||||
function](https://github.com/thallada/nlp/blob/master/syntax_aware_generate.py)).
|
||||
|
||||
```python
|
||||
from syntax_aware_generate import generate
|
||||
|
||||
# inserts matching syntax subtrees from trump.txt into
|
||||
# trees from austen-emma.txt
|
||||
generate('trump.txt', word_limit=10)
|
||||
```
|
||||
```
|
||||
(SBARQ
|
||||
(SQ
|
||||
(NP (PRP I))
|
||||
(VP (VBP do) (RB not) (VB advise) (NP (DT the) (NN custard))))
|
||||
(. .))
|
||||
I do not advise the custard .
|
||||
==============================
|
||||
I do n't want the drone !
|
||||
(SBARQ
|
||||
(SQ
|
||||
(NP (PRP I))
|
||||
(VP (VBP do) (RB n't) (VB want) (NP (DT the) (NN drone))))
|
||||
(. !))
|
||||
```
|
||||
|
||||
Above the line is a sentence selected from a corpus of Jane Austen's *Emma*.
|
||||
Below it is a sentence generated by walking down the syntax tree and finding
|
||||
sub-trees from a corpus of Trump's tweets that match the same syntactical
|
||||
structure and then swapping the words in.
|
||||
|
||||
The result can sometimes be amusing, but more often than not, this approach
|
||||
doesn't fare much better than the n-gram based generation.
|
||||
|
||||
### spaCy
|
||||
|
||||
I'm only beginning to experiment with the [spaCy](https://spacy.io/) Python
|
||||
library, but I like it a lot. For one, it is much, much faster than NLTK:
|
||||
|
||||
|
||||
|
||||
[https://spacy.io/docs/api/#speed-comparison](https://spacy.io/docs/api/#speed-comparison)
|
||||
|
||||
The [API](https://spacy.io/docs/api/) takes a little getting used to coming from
|
||||
NLTK. It doesn't seem to have any sort of out-of-the-box solution to printing
|
||||
out syntax trees like above, but it does do [part-of-speech
|
||||
tagging](https://spacy.io/docs/api/tagger) and [dependency relation
|
||||
mapping](https://spacy.io/docs/api/dependencyparser) which should accomplish
|
||||
about the same. You can see both of these visually with
|
||||
[displaCy](https://demos.explosion.ai/displacy/).
|
||||
|
||||
## Neural Network Based Generation
|
||||
|
||||
If you haven't heard all the buzz about [neural
|
||||
networks](https://en.wikipedia.org/wiki/Artificial_neural_network), they are a
|
||||
particular technique for [machine
|
||||
learning](https://en.wikipedia.org/wiki/Machine_learning) that's inspired by our
|
||||
understanding of the human brain. They are structured into layers of nodes which
|
||||
have connections to other nodes in other layers of the network. These
|
||||
connections have weights which each node multiplies by the corresponding input
|
||||
and enters into a particular [activation
|
||||
function](https://en.wikipedia.org/wiki/Activation_function) to output a single
|
||||
number. The optimal weights for solving a particular problem with the network
|
||||
are learned by training the network using
|
||||
[backpropagation](https://en.wikipedia.org/wiki/Backpropagation) to perform
|
||||
[gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) on a
|
||||
particular [cost function](https://en.wikipedia.org/wiki/Loss_function) that
|
||||
tries to balance getting the correct answer while also
|
||||
[generalizing](https://en.wikipedia.org/wiki/Regularization_(mathematics)) the
|
||||
network enough to perform well on data the network hasn't seen before.
|
||||
|
||||
[Long short-term memory
|
||||
(LSTM)](https://en.wikipedia.org/wiki/Long_short-term_memory) is a type of
|
||||
[recurrent neural network
|
||||
(RNN)](https://en.wikipedia.org/wiki/Recurrent_neural_network) (a network with
|
||||
cycles) that can remember previous values for a short or long period of time.
|
||||
This property makes them remarkably effective at a multitude of tasks, one of
|
||||
which is predicting text that will follow a given sequence. We can use this to
|
||||
continually generate text by inputting a seed, appending the generated output to
|
||||
the end of the seed, removing the first element from the beginning of the seed,
|
||||
and then inputting the seed again, following the same process until we've
|
||||
generated enough text from the network ([paper on using RNNs to generate
|
||||
text](http://www.cs.utoronto.ca/~ilya/pubs/2011/LANG-RNN.pdf)).
|
||||
|
||||
Luckily, a lot of smart people have done most of the legwork so you can just
|
||||
download their neural network architecture and train it yourself. There's
|
||||
[char-rnn](https://github.com/karpathy/char-rnn) which has some [really exciting
|
||||
results for generating texts (e.g. fake
|
||||
Shakespeare)](http://karpathy.github.io/2015/05/21/rnn-effectiveness/). There's
|
||||
also [word-rnn](https://github.com/larspars/word-rnn) which is a modified
|
||||
version of char-rnn that operates on words as a unit instead of characters.
|
||||
Follow [my last blog post on how to install TensorFlow on Ubuntu
|
||||
16.04](/2017/06/20/how-to-install-tensorflow-on-ubuntu-16-04.html) and
|
||||
you'll be almost ready to run a TensorFlow port of word-rnn:
|
||||
[word-rnn-tensorflow](https://github.com/hunkim/word-rnn-tensorflow).
|
||||
|
||||
I plan on playing around with NNs a lot more to see what kind of poetry-looking
|
||||
text I can generate from them.
|
||||
|
||||
---
|
||||
|
||||
[^1]:
|
||||
Fun fact: They used to be pronounced differently in Middle English during
|
||||
the invention of the printing press and standardized spelling. The [Great
|
||||
Vowel Shift](https://en.wikipedia.org/wiki/Great_Vowel_Shift) happened
|
||||
after, and is why they are now pronounced the same.
|
||||
@ -1,76 +0,0 @@
|
||||
---
|
||||
title: "Proximity Structures: Playing around with PixiJS"
|
||||
layout: post
|
||||
image: /img/blog/proximity-structures.png
|
||||
---
|
||||
|
||||
I've been messing around with a library called [PixiJS](http://www.pixijs.com/)
|
||||
which allows you to create WebGL animations which will fall back to HTML5 canvas
|
||||
if WebGL is not available in the browser. I mostly like it because the API is
|
||||
similar to HTML5 canvas which [I was already familiar
|
||||
with](https://github.com/thallada/thallada.github.io/blob/master/js/magic.js). I
|
||||
can't say that I like the PixiJS API and documentation that much, though. For
|
||||
this project, I mostly just used a small portion of it to create [WebGL (GPU
|
||||
accelerated) primitive
|
||||
shapes](http://www.goodboydigital.com/pixi-webgl-primitives/) (lines and
|
||||
circles).
|
||||
<!--excerpt-->
|
||||
|
||||
**Play with it here**: [http://proximity.hallada.net](http://proximity.hallada.net)
|
||||
|
||||
**Read/clone the code here**: [https://github.com/thallada/proximity-structures](https://github.com/thallada/proximity-structures)
|
||||
|
||||
[](http://proximity.hallada.net)
|
||||
|
||||
The idea was inspired by
|
||||
[all](https://thumb9.shutterstock.com/display_pic_with_logo/3217643/418838422/stock-vector-abstract-technology-futuristic-network-418838422.jpg)
|
||||
[those](https://ak5.picdn.net/shutterstock/videos/27007555/thumb/10.jpg)
|
||||
[countless](https://ak9.picdn.net/shutterstock/videos/10477484/thumb/1.jpg)
|
||||
[node](https://ak3.picdn.net/shutterstock/videos/25825727/thumb/1.jpg)
|
||||
[network](https://t4.ftcdn.net/jpg/00/93/24/21/500_F_93242102_mqtDljufY7CNY0wMxunSbyDi23yNs1DU.jpg)
|
||||
[graphics](https://ak6.picdn.net/shutterstock/videos/12997085/thumb/1.jpg) that
|
||||
I see all the time as stock graphics on generic tech articles.
|
||||
|
||||
This was really fun to program. I didn't care much about perfect code, I just
|
||||
kept hacking one thing onto another while watching the instantaneous feedback of
|
||||
the points and lines responding to my changes until I had something worth
|
||||
sharing.
|
||||
|
||||
### Details
|
||||
|
||||
The majority of the animation you see is based on
|
||||
[tweening](https://en.wikipedia.org/wiki/Inbetweening). Each point has an origin
|
||||
and destination stored in memory. Every clock tick (orchestrated by the almighty
|
||||
[requestAnimationFrame](https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame)),
|
||||
the main loop calculates where each point should be in the path between its
|
||||
origin and destination based on how long until it completes its "cycle". There
|
||||
is a global `cycleDuration`, defaulted to 60. Every frame increments the cycle
|
||||
counter by 1 until it reaches 60, at which point it folds over back to 0. Every
|
||||
point is assigned a number between 1 and 60. This is its start cycle. When the
|
||||
global cycle counter equals a point's start cycle number, the point has reached
|
||||
its destination and a new target destination is randomly chosen.
|
||||
|
||||
Each point is also randomly assigned a color. When a point is within
|
||||
`connectionDistance` of another point in the canvas, a line is drawn between the
|
||||
two points, their colors are averaged, and the points' colors become the average
|
||||
color weighted by the distance between the points. You can see clusters of
|
||||
points converging on a color in the animation.
|
||||
|
||||
Click interaction is implemented by modifying point target destinations within a
|
||||
radius around the click. Initially, a mouse hover will push points away.
|
||||
Clicking and holding will draw points in, progressively growing the effect
|
||||
radius in the process to capture more and more points.
|
||||
|
||||
I thought it was really neat that without integrating any physics engine
|
||||
whatsoever, I ended up with something that looked sort of physics based thanks
|
||||
to the tweening functions. Changing the tweening functions that the points use
|
||||
seems to change the physical properties and interactions of the points. The
|
||||
elastic tweening function makes the connections between the points snap like
|
||||
rubber bands. And, while I am not drawing any explicit polygons, just points and
|
||||
lines based on proximity, it sometimes looks like the points are coalescing into
|
||||
some three-dimensional structure.
|
||||
|
||||
I'll probably make another procedural animation like this in the future since it
|
||||
was so fun. Next time, I'll probably start from the get-go in ES2015 (or ES7,
|
||||
or ES8??) and proper data structures.
|
||||
@ -1,185 +0,0 @@
|
||||
---
|
||||
title: "Making a Mailing List for a Jekyll Blog Using Sendy"
|
||||
layout: post
|
||||
---
|
||||
|
||||
When my beloved [Google Reader](https://en.wikipedia.org/wiki/Google_Reader) was
|
||||
discontinued in 2013, I stopped regularly checking RSS feeds. Apparently, [I am
|
||||
not alone](https://trends.google.com/trends/explore?date=all&q=rss). It seems
|
||||
like there's a new article every month arguing either that [RSS is dead or RSS
|
||||
is not dead
|
||||
yet](https://hn.algolia.com/?q=&query=rss%20dead&sort=byPopularity&prefix&page=0&dateRange=all&type=story).
|
||||
Maybe RSS will stick around to serve as a cross-site communication backbone, but
|
||||
I don't think anyone will refute that RSS feeds are declining in consumer use.
|
||||
Facebook, Twitter, and other aggregators are where people really go. However, I
|
||||
noticed that I still follow some small infrequent blogs through mailing lists
|
||||
that they offer. I'm really happy to see an email sign up on blogs I like,
|
||||
because it means I'll know when they post new content in the future. I check my
|
||||
email regularly unlike my RSS feeds.
|
||||
<!--excerpt-->
|
||||
|
||||
Even though I'm sure my blog is still too uninteresting and unheard of to get
|
||||
many signups, I still wanted to know what it took to make a blog mailing list.
|
||||
RSS is super simple for website owners, because all they need to do is dump all
|
||||
of their content into a specially formatted XML file, host it, and let RSS
|
||||
readers deal with all the complexity. In my blog, [I didn't even need a
|
||||
Jekyll
|
||||
plugin](https://github.com/thallada/thallada.github.io/blob/master/feed.xml).
|
||||
Email is significantly more difficult. With email, the website owner owns more
|
||||
of the complexity. And, spam filters make it unfeasible to roll your own email
|
||||
server. A couple people can mark you as spam, and BAM: now you are blacklisted
|
||||
and you have to move to a new IP address. This is why most people turn to a
|
||||
hosted service like [Mailchimp](https://mailchimp.com/). Though, I was
|
||||
dissatisfied with that because of the [high costs and measly free
|
||||
tier](https://mailchimp.com/pricing/).
|
||||
|
||||
[Amazon Simple Email Service (SES)](https://aws.amazon.com/ses/) deals with all
|
||||
the complexity of email for you and is also
|
||||
[cheap](https://aws.amazon.com/ses/pricing/). In fact, it's free unless you have
|
||||
more than 62,000 subscribers or post way more than around once a month, and even
|
||||
after that it's a dime for every 1,000 emails sent. Frankly, no one can really
|
||||
compete with what Amazon is offering here.
|
||||
|
||||
Okay, so that covers sending the emails, but what about collecting and storing
|
||||
subscriptions? SES doesn't handle any of that. I searched around a long time for
|
||||
something simple and free that wouldn't require me setting up a server [^1]. I
|
||||
eventually ended up going with [Sendy](https://sendy.co/) because it looked like
|
||||
a well-designed product exactly for this use case that also handled drafting
|
||||
emails, email templates, confirmation emails, and analytics. It costs a one-time
|
||||
fee of $59 and I was willing to fork that over for quality software. Especially
|
||||
since most other email newsletter services require some sort of monthly
|
||||
subscription that scales with the number of emails you are sending.
|
||||
|
||||
Unfortunately, since Sendy is self-hosted, I had to add a dynamic server to my
|
||||
otherwise completely static Jekyll website hosted for free on [Github
|
||||
Pages](https://pages.github.com/). You can put Sendy on pretty much anything
|
||||
that runs PHP and MySQL including the cheap [t2.micro Amazon EC2 instance
|
||||
type](https://aws.amazon.com/ec2/instance-types/). If you are clever, you might
|
||||
find a cheaper way. I already had a t2.medium for general development,
|
||||
tinkering, and hosting, so I just used that.
|
||||
|
||||
There are many guides out there for setting up MySQL and Apache, so I won't go
|
||||
over that. But, I do want to mention how I got Sendy to integrate with
|
||||
[nginx](https://nginx.org/en/) which is the server engine I was already using. I
|
||||
like to put separate services I'm running under different subdomains of
|
||||
my domain hallada.net even though they are running on the same server and IP
|
||||
address. For Sendy, I chose [list.hallada.net](http://list.hallada.net) [^2].
|
||||
Setting up another subdomain in nginx requires [creating a new server
|
||||
block](https://askubuntu.com/a/766369). There's [a great Gist of a config for
|
||||
powering Sendy using nginx and
|
||||
FastCGI](https://gist.github.com/refringe/6545132), but I ran into so many
|
||||
issues with the subdomain that I decided to use nginx as a proxy to the Apache
|
||||
mod_php site running Sendy. I'll just post my config here:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
|
||||
server_name list.hallada.net;
|
||||
|
||||
root /var/www/html/sendy;
|
||||
index index.php;
|
||||
|
||||
location /l/ {
|
||||
rewrite ^/l/([a-zA-Z0-9/]+)$ /l.php?i=$1 last;
|
||||
}
|
||||
|
||||
location /t/ {
|
||||
rewrite ^/t/([a-zA-Z0-9/]+)$ /t.php?i=$1 last;
|
||||
}
|
||||
|
||||
location /w/ {
|
||||
rewrite ^/w/([a-zA-Z0-9/]+)$ /w.php?i=$1 last;
|
||||
}
|
||||
|
||||
location /unsubscribe/ {
|
||||
rewrite ^/unsubscribe/(.*)$ /unsubscribe.php?i=$1 last;
|
||||
}
|
||||
|
||||
location /subscribe/ {
|
||||
rewrite ^/subscribe/(.*)$ /subscribe.php?i=$1 last;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_set_header Host $host;
|
||||
proxy_pass http://127.0.0.1:8080/sendy/;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Basically, this proxies all of the requests through to Apache which I configured
|
||||
to run on port 8080 by changing the `Listen` directive in
|
||||
`/etc/apache2/ports.conf`.
|
||||
|
||||
I also had to add `RewriteBase /sendy` to the end of the `.htcaccess` file in
|
||||
the sendy directory (which, for me, was in `/var/www/html/sendy`). This
|
||||
basically forces Sendy to use urls that start with `http://list.hallada.net`
|
||||
instead of `http://list.hallada.net/sendy` which I thought was redundant since I
|
||||
am dedicating the whole subdomain to sendy.
|
||||
|
||||
A perplexing issue I ran into was that Gmail accounts were completely dropping
|
||||
(not even bouncing!) any emails I sent to them if I used my personal email
|
||||
`tyler@hallada.net` as the from address. I switched to `tyhallada@gmail.com` for
|
||||
the from address and emails went through fine after that [^4]. [The issue seems
|
||||
unresolved](https://forums.aws.amazon.com/thread.jspa?messageID=802461󃺝)
|
||||
as of this post.
|
||||
|
||||
Lastly, I needed to create a form on my website for readers to sign up for the
|
||||
mailing list. Sendy provides the HTML in the UI to create the form, which I
|
||||
[tweaked a
|
||||
little](https://github.com/thallada/thallada.github.io/blob/master/_includes/mail-form.html)
|
||||
and placed in a [Jekyll includes template
|
||||
partial](https://jekyllrb.com/docs/includes/) that I could include on both the
|
||||
post layout and the blog index template. I refuse to pollute the internet with
|
||||
yet another annoying email newsletter form that pops up while you are trying to
|
||||
read the article, so you can find my current version at the bottom of this
|
||||
article where it belongs [^5].
|
||||
|
||||
All in all, setting up a mailing list this way wasn't too bad except for the part
|
||||
where I spent way too much time fiddling with nginx configs. But, I always do
|
||||
that, so I guess that's expected.
|
||||
|
||||
As for the content of the newsletter, I haven't figured out how to post the
|
||||
entirety of a blog post into the HTML format of an email as soon as I commit a
|
||||
new post yet. So, I think for now I will just manually create a new email
|
||||
campaign in Sendy (from an email template) that will have a link to the new
|
||||
post, and send that.
|
||||
|
||||
---
|
||||
|
||||
[^1]:
|
||||
It would be interesting to look into creating a [Google
|
||||
Form](https://www.google.com/forms/about/) that submits rows to a [Google
|
||||
Sheet](https://www.google.com/sheets/about/) and then triggering a [AWS
|
||||
Lambda](https://aws.amazon.com/lambda/) service that iterates over the rows
|
||||
using something like [the Google Sheets Python
|
||||
API](https://developers.google.com/sheets/api/quickstart/python) and sending
|
||||
an email for every user using the [Amazon SES
|
||||
API](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-api.html)
|
||||
([python-amazon-ses-api](https://github.com/pankratiev/python-amazon-ses-api)
|
||||
might also be useful there).
|
||||
|
||||
[^2]:
|
||||
I ran into a hiccup [verifying this domain for Amazon
|
||||
SES](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-domain-procedure.html)
|
||||
using the [Namecheap](https://www.namecheap.com/) advanced DNS settings
|
||||
because it only allowed me to set up one MX record, but I already had one
|
||||
for my root hallada.net domain that I needed. So, I moved to [Amazon's Route
|
||||
53](https://aws.amazon.com/route53/) instead [^3] which made setting up the
|
||||
[DKIM
|
||||
verification](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/easy-dkim.html)
|
||||
really easy since Amazon SES gave a button to create the necessary DNS
|
||||
records directly in my Route 53 account.
|
||||
|
||||
[^3]:
|
||||
As [Amazon continues its plan for world
|
||||
domination](https://www.washingtonpost.com/business/is-amazon-getting-too-big/2017/07/28/ff38b9ca-722e-11e7-9eac-d56bd5568db8_story.html)
|
||||
it appears I'm moving more and more of my personal infrastructure over to
|
||||
Amazon as well...
|
||||
|
||||
[^4]: Obviously a conspiracy by Google to force domination of Gmail.
|
||||
|
||||
[^5]: Yes, I really hate those pop-ups.
|
||||
@ -1,259 +0,0 @@
|
||||
---
|
||||
title: "Isso Comments"
|
||||
layout: post
|
||||
---
|
||||
|
||||
I've been meaning to add a commenting system to this blog for a while, but I
|
||||
couldn't think of a good way to do it. I implemented my own commenting system on
|
||||
my [old Django personal site](https://github.com/thallada/personalsite). While I
|
||||
enjoyed working on it at the time, it was a lot of work, especially to fight the
|
||||
spam. Now that my blog is hosted statically on Github's servers, I have no way
|
||||
to host something dynamic like comments.
|
||||
<!--excerpt-->
|
||||
|
||||
[Disqus](http://disqus.com/) seems to be the popular solution to this problem
|
||||
for other people that host static blogs. The way it works is that you serve a
|
||||
javascript client script on the static site you own. The script will make AJAX
|
||||
requests to a separate server that Disqus owns to retrieve comments and post new
|
||||
ones.
|
||||
|
||||
The price you pay for using Disqus, however, is that [they get to sell all of
|
||||
the data that you and your commenters give
|
||||
them](https://replyable.com/2017/03/disqus-is-your-data-worth-trading-for-convenience/).
|
||||
That reason, plus the fact that I wanted something more DIY, meant this blog has
|
||||
gone without comments for a few years.
|
||||
|
||||
Then I discovered [Isso](https://github.com/posativ/isso). Isso calls itself a
|
||||
lightweight alternative to [Disqus](http://disqus.com/). Isso allows you to
|
||||
install the server code on your own server so that the comment data never goes
|
||||
to a third party. Also, it does not require logging into some social media
|
||||
account just to comment. Today, I installed it on my personal AWS EC2 instance
|
||||
and added the Isso javascript client script on this blog. So far, my experience
|
||||
with it has been great and it performs exactly the way I expect.
|
||||
|
||||
I hit a few snags while installing it, however.
|
||||
|
||||
## Debian Package
|
||||
|
||||
**I don't recommend using the Debian package anymore as it frequently goes out
|
||||
of date and breaks on distribution upgrades. See bottom edit.**
|
||||
|
||||
There is a very handy [Debian package](https://github.com/jgraichen/debian-isso)
|
||||
that someone has made for Isso. Since my server runs Ubuntu 16.04, and Ubuntu is
|
||||
based off of Debian, this is a package I can install with my normal ubuntu
|
||||
package manager utilities. There is no PPA to install since the package is in
|
||||
the [main Ubuntu package archive](https://packages.ubuntu.com/xenial/isso). Just
|
||||
run `sudo apt-get install isso`.
|
||||
|
||||
I got a bit confused after that point, though. There seems to be no
|
||||
documentation I could find about how to actually configure and start the server
|
||||
once you have installed it. This is what I did:
|
||||
|
||||
```bash
|
||||
sudo cp /etc/default/isso /etc/isso.d/available/isso.cfg
|
||||
sudo ln -s /etc/isso.d/available/isso.cfg /etc/isso.d/enabled/isso.cfg
|
||||
```
|
||||
|
||||
Then you can edit `/etc/isso.d/available/isso.cfg` with your editor of choice to
|
||||
[configure the Isso server for your
|
||||
needs](https://posativ.org/isso/docs/configuration/server/). Make sure to set
|
||||
the `host` variable to the URL for your static site.
|
||||
|
||||
Once you're done, you can run `sudo service isso restart` to reload the server
|
||||
with the new configuration. `sudo service isso status` should report `Active
|
||||
(running)`.
|
||||
|
||||
Right now, there should be a [gunicorn](http://gunicorn.org/) process running
|
||||
the isso server. You can check that with `top` or running `ps aux | grep
|
||||
gunicorn`, which should return something about "isso".
|
||||
|
||||
## Nginx Reverse Proxy
|
||||
|
||||
In order to map the URL "comments.hallada.net" to this new gunicorn server, I
|
||||
need an [nginx reverse
|
||||
proxy](https://www.nginx.com/resources/admin-guide/reverse-proxy/).
|
||||
|
||||
To do that, I made a new server block: `sudo vim
|
||||
/etc/nginx/sites-available/isso` which I added:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name comments.hallada.net;
|
||||
|
||||
location / {
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Script-Name /isso;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_pass http://localhost:8000;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then I enabled this new server block with:
|
||||
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/isso /etc/nginx/sites-enabled/isso
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
## DNS Configuration
|
||||
|
||||
I added a new A record for "comments.hallada.net" that pointed to my server's IP
|
||||
address to the DNS configuration for my domain (which I recently switched to
|
||||
[Amazon Route 53](https://aws.amazon.com/route53/)).
|
||||
|
||||
After the DNS caches had time to refresh, visiting `http://comments.hallada.net`
|
||||
would hit the new `isso` nginx server block, which would then pass the request
|
||||
on to the gunicorn process.
|
||||
|
||||
You can verify if nginx is getting the request by looking at
|
||||
`/var/log/nginx/access.log`.
|
||||
|
||||
## Adding the Isso Script to my Jekyll Site
|
||||
|
||||
I created a file called `_includes/comments.html` with the contents that [the
|
||||
Isso documentation](https://posativ.org/isso/docs/quickstart/#integration)
|
||||
provides. Then, in my post template, I simply included that on the page where I
|
||||
wanted the comments to go:
|
||||
|
||||
```html
|
||||
{% include comments.html %}
|
||||
```
|
||||
|
||||
Another thing that was not immediately obvious to me is that the value of the
|
||||
`name` variable in the Isso server configuration is the URL path that you will
|
||||
need to point the Isso JavaScript client to. For example, I chose `name = blog`,
|
||||
so the `data-isso` attribute on the script tag needed to be
|
||||
`http://comments.hallada.net/blog/`.
|
||||
|
||||
## The Uncaught ReferenceError
|
||||
|
||||
**You won't need to fix this if you install Isso from PIP! See bottom edit.**
|
||||
|
||||
There's [an issue](https://github.com/posativ/isso/issues/318) with that Debian
|
||||
package that causes a JavaScript error in the console when trying to load the
|
||||
Isso script in the browser. I solved this by uploading the latest version of the
|
||||
Isso `embeded.min.js` file to my server, which I put at
|
||||
`/var/www/html/isso/embeded.min.js`. Then I modified the nginx server block to
|
||||
serve that file when the path matches `/isso`:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name comments.hallada.net;
|
||||
root /var/www/html;
|
||||
|
||||
location / {
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Script-Name /isso;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_pass http://localhost:8000;
|
||||
}
|
||||
|
||||
location /isso {
|
||||
try_files $uri $uri/ $uri.php?$args =404;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Now requesting `http://comments.hallada.net/isso/embeded.min.js` would return
|
||||
the newer script without the bug.
|
||||
|
||||
## Sending Emails Through Amazon Simple Email Service
|
||||
|
||||
I already set up [Amazon's SES](https://aws.amazon.com/ses/) in my [last
|
||||
blog
|
||||
post](http://www.hallada.net/2017/08/30/making-mailing-list-jekyll-blog-using-sendy.html).
|
||||
To get Isso to use SES to send notifications about new comments, create a new
|
||||
credential in the SES UI, and then set the `user` and `password` fields in the
|
||||
`isso.cfg` to what get's generated for the IAM user. The SES page also has
|
||||
information for what `host` and `port` to use. I used `security = starttls` and
|
||||
`port = 587`. Make sure whatever email you use for `from` is a verified email in
|
||||
SES. Also, don't forget to add your email as the `to` value.
|
||||
|
||||
## Enabling HTTPS with Let's Encrypt
|
||||
|
||||
[Let's Encrypt](https://letsencrypt.org/) allows you to get SSL certificates for
|
||||
free! I had already installed the certbot/letsencrypt client before, so I just
|
||||
ran this to generate a new certificate for my new sub-domain
|
||||
"comments.hallada.net":
|
||||
|
||||
```bash
|
||||
sudo letsencrypt certonly --nginx -d comments.hallada.net
|
||||
```
|
||||
|
||||
Once that successfully completed, I added a new nginx server block for the https
|
||||
version at `/etc/nginx/sites-available/isso-https`:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
listen [::]:443 ssl http2;
|
||||
server_name comments.hallada.net;
|
||||
root /var/www/html;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/comments.hallada.net/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/comments.hallada.net/privkey.pem;
|
||||
ssl_trusted_certificate /etc/letsencrypt/live/comments.hallada.net/fullchain.pem;
|
||||
|
||||
location / {
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Script-Name /isso;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_pass http://localhost:8000;
|
||||
}
|
||||
|
||||
location /isso {
|
||||
try_files $uri $uri/ $uri.php?$args =404;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
And, I changed the old http server block so that it just permanently redirects
|
||||
to the https version:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name comments.hallada.net;
|
||||
root /var/www/html;
|
||||
|
||||
location / {
|
||||
return 301 https://comments.hallada.net$request_uri;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then I enabled the https version:
|
||||
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/isso-https /etc/nginx/sites-enabled/isso-https
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
I checked that I didn't get any errors visiting `https://comments.hallada.net/`,
|
||||
and then changed my Jekyll include snippet so that it pointed at the `https`
|
||||
site instead of `http`.
|
||||
|
||||
Now you can securely leave a comment if you want to yell at me for writing the
|
||||
wrong thing!
|
||||
|
||||
## EDIT 5/28/2019:
|
||||
|
||||
I don't recommend using the Debian package anymore since it frequently goes out
|
||||
of date and breaks when upgrading your Linux distribution.
|
||||
|
||||
Instead, follow the [Isso docs](https://posativ.org/isso/docs/install/) by
|
||||
creating a [virtualenv](https://virtualenv.pypa.io/en/latest/) and then run `pip
|
||||
install isso` and `pip install gunicorn` from within the virtualenv. Then, when
|
||||
creating [a systemd
|
||||
service](https://github.com/jgraichen/debian-isso/blob/master/debian/isso.service),
|
||||
make sure to point to the gunicorn executable in that virtualenv (e.g.
|
||||
`/opt/isso/bin/gunicorn`). It should load and run Isso from the same virtualenv.
|
||||
@ -1,365 +0,0 @@
|
||||
---
|
||||
title: "Studio-Frontend: Developing Frontend Separate from edX Platform"
|
||||
layout: post
|
||||
---
|
||||
|
||||
*This is a blog post that I originally wrote for the [edX engineering
|
||||
blog](https://engineering.edx.org/).*
|
||||
|
||||
At the core of edX is the [edx-platform](https://github.com/edx/edx-platform), a
|
||||
monolithic Django code-base 2.7 times the size of Django itself.
|
||||
<!--excerpt-->
|
||||
|
||||
```
|
||||
-------------------------------------------------------------------------------
|
||||
Language Files Lines Code Comments Blanks
|
||||
-------------------------------------------------------------------------------
|
||||
ActionScript 1 118 74 23 21
|
||||
Autoconf 10 425 237 163 25
|
||||
CSS 55 17106 14636 1104 1366
|
||||
HTML 668 72567 36865 30306 5396
|
||||
JavaScript 1500 463147 352306 55882 54959
|
||||
JSON 91 14583 14583 0 0
|
||||
JSX 33 2595 2209 62 324
|
||||
LESS 1 949 606 232 111
|
||||
Makefile 1 65 49 8 8
|
||||
Markdown 23 287 287 0 0
|
||||
Mustache 1 1 1 0 0
|
||||
Python 3277 559255 442756 29254 87245
|
||||
ReStructuredText 48 4252 4252 0 0
|
||||
Sass 424 75559 55569 4555 15435
|
||||
Shell 15 929 505 292 132
|
||||
SQL 4 6283 5081 1186 16
|
||||
Plain Text 148 3521 3521 0 0
|
||||
TypeScript 20 88506 76800 11381 325
|
||||
XML 364 5283 4757 231 295
|
||||
YAML 36 1630 1361 119 150
|
||||
-------------------------------------------------------------------------------
|
||||
Total 6720 1317061 1016455 134798 165808
|
||||
-------------------------------------------------------------------------------
|
||||
```
|
||||
|
||||
35% of the edx-platform is JavaScript. While it has served edX well since its
|
||||
inception in 2012, reaching over 11 million learners in thousands of courses on
|
||||
[edX.org](https://www.edx.org/) and many more millions on all of the [Open edX
|
||||
instances across the
|
||||
world](https://openedx.atlassian.net/wiki/spaces/COMM/pages/162245773/Sites+powered+by+Open+edX),
|
||||
it is starting to show its age. Most of it comes in the form of [Backbone.js
|
||||
apps](http://backbonejs.org/) loaded by [RequireJS](http://requirejs.org/) in
|
||||
Django [Mako templates](http://www.makotemplates.org/), with
|
||||
[jQuery](https://jquery.com/) peppered throughout.
|
||||
|
||||
Many valiant efforts are underway to modernize the frontend of edx-platform
|
||||
including replacing RequireJS with Webpack, Backbone.js with
|
||||
[React](https://reactjs.org/), and ES5 JavaScript and CoffeeScript with ES6
|
||||
JavaScript. Many of these efforts [were covered in detail at the last Open edX
|
||||
conference](https://www.youtube.com/watch?v=xicBnbDX4AY) and in [Open edX
|
||||
Proposal 11: Front End Technology
|
||||
Standards](https://open-edx-proposals.readthedocs.io/en/latest/oep-0011-bp-FED-technology.html).
|
||||
However, the size and complexity of the edx-platform means that these kind of
|
||||
efforts are hard to prioritize, and, in the meantime, frontend developers are
|
||||
forced to [wait over 10
|
||||
minutes](https://openedx.atlassian.net/wiki/spaces/FEDX/pages/264700138/Asset+Compilation+Audit+2017-11-01)
|
||||
for our home-grown asset pipeline to build before they can view changes.
|
||||
|
||||
There have also been efforts to incrementally modularize and extract parts of
|
||||
the edx-platform into separate python packages that could be installed as
|
||||
[Django apps](https://docs.djangoproject.com/en/2.0/ref/applications/), or even
|
||||
as separately deployed
|
||||
[microservices](https://en.wikipedia.org/wiki/Microservices). This allows
|
||||
developers to work independently from the rest of the organization inside of a
|
||||
repository that they own, manage, and is small enough that they could feasibly
|
||||
understand it entirely.
|
||||
|
||||
When my team was tasked with improving the user experience of pages in
|
||||
[Studio](https://studio.edx.org/), the tool that course authors use to create
|
||||
course content, we opted to take a similar architectural approach with the
|
||||
frontend and create a new repository where we could develop new pages in
|
||||
isolation and then integrate them back into the edx-platform as a plugin. We
|
||||
named this new independent repository
|
||||
[studio-frontend](https://github.com/edx/studio-frontend). With this approach,
|
||||
our team owns the entire studio-frontend code-base and can make the best
|
||||
architectural changes required for its features without having to consult with
|
||||
and contend with all of the other teams at edX that contribute to the
|
||||
edx-platform. Developers of studio-frontend can also avoid the platform’s slow
|
||||
asset pipeline by doing all development within the studio-frontend repository
|
||||
and then later integrating the changes into platform.
|
||||
|
||||
## React and Paragon
|
||||
|
||||
When edX recently started to conform our platform to the [Web Content
|
||||
Accessibility Guidelines 2.0 AA (WCAG 2.0
|
||||
AA)](https://www.w3.org/WAI/intro/wcag), we faced many challenges in
|
||||
retrofitting our existing frontend code to be accessible. Rebuilding Studio
|
||||
pages from scratch in studio-frontend allows us to not only follow the latest
|
||||
industry standards for building robust and performant frontend applications, but
|
||||
to also build with accessibility in mind from the beginning.
|
||||
|
||||
The Javascript community has made great strides recently to [address
|
||||
accessibility issues in modern web
|
||||
apps](https://reactjs.org/docs/accessibility.html). However, we had trouble
|
||||
finding an open-source React component library that fully conformed to WCAG 2.0
|
||||
AA and met all of edX’s needs, so we decided to build our own:
|
||||
[Paragon](https://github.com/edx/paragon).
|
||||
|
||||
Paragon is a library of building-block components like buttons, inputs, icons,
|
||||
and tables which were built from scratch in React to be accessible. The
|
||||
components are styled using the [Open edX theme of Bootstrap
|
||||
v4](https://github.com/edx/edx-bootstrap) (edX’s decision to adopt Bootstrap is
|
||||
covered in
|
||||
[OEP-16](https://open-edx-proposals.readthedocs.io/en/latest/oep-0016-bp-adopt-bootstrap.html)).
|
||||
Users of Paragon may also choose to use the
|
||||
[themeable](https://github.com/edx/paragon#export-targets) unstyled target and
|
||||
provide their own Bootstrap theme.
|
||||
|
||||
|
||||
|
||||
|
||||
Studio-frontend composes together Paragon components into higher-level
|
||||
components like [an accessibility
|
||||
form](https://github.com/edx/studio-frontend/blob/master/src/accessibilityIndex.jsx)
|
||||
or [a table for course assets with searching, filtering, sorting, pagination,
|
||||
and upload](https://github.com/edx/studio-frontend/blob/master/src/index.jsx).
|
||||
While we developed these components in studio-frontend, we were able to improve
|
||||
the base Paragon components. Other teams at edX using the same components were
|
||||
able to receive the same improvements with a single package update.
|
||||
|
||||
|
||||
|
||||
## Integration with Studio
|
||||
|
||||
We were able to follow the typical best practices for developing a React/Redux
|
||||
application inside studio-frontend, but at the end of the day, we still had to
|
||||
somehow get our components inside of existing Studio pages and this is where
|
||||
most of the challenges arose.
|
||||
|
||||
## Webpack
|
||||
|
||||
The aforementioned move from RequireJS to Webpack in the edx-platform made it
|
||||
possible for us to build our studio-frontend components from source with Webpack
|
||||
within edx-platform. However, this approach tied us to the edx-platform’s slow
|
||||
asset pipeline. If we wanted rapid development, we had to duplicate the
|
||||
necessary Webpack config between both studio-frontend and edx-platform.
|
||||
|
||||
Instead, studio-frontend handles building the development and production Webpack
|
||||
builds itself. In development mode, the incremental rebuild that happens
|
||||
automatically when a file is changed takes under a second. The production
|
||||
JavaScript and CSS bundles, which take about 25 seconds to build, are published
|
||||
with every new release to
|
||||
[NPM](https://www.npmjs.com/package/@edx%2Fstudio-frontend). The edx-platform
|
||||
`npm install`s studio-frontend and then copies the built production files from
|
||||
`node_modules` into its Django static files directory where the rest of the
|
||||
asset pipeline will pick it up.
|
||||
|
||||
To actually use the built JavaScript and CSS, edx-platform still needs to
|
||||
include it in its Mako templates. We made a [Mako template
|
||||
tag](https://github.com/edx/edx-platform/blob/master/common/djangoapps/pipeline_mako/templates/static_content.html#L93-L122)
|
||||
that takes a Webpack entry point name in studio-frontend and generates script
|
||||
tags that include the necessary files from the studio-frontend package. It also
|
||||
dumps all of the initial context that studio-frontend needs from the
|
||||
edx-platform Django app into [a JSON
|
||||
object](https://github.com/edx/edx-platform/blob/master/cms/templates/asset_index.html#L36-L56)
|
||||
in a script tag on the page that studio-frontend components can access via a
|
||||
shared id. This is how studio-frontend components get initial data from Studio,
|
||||
like which course it’s embedded in.
|
||||
|
||||
For performance, modules that are shared across all studio-frontend components
|
||||
are extracted into `common.min.js` and `common.min.css` files that are included
|
||||
on every Studio template that has a studio-frontend component. User's browsers
|
||||
should cache these files so that they do not have to re-download libraries like
|
||||
React and Redux every time they visit a new page that contains a studio-frontend
|
||||
component.
|
||||
|
||||
## CSS Isolation
|
||||
|
||||
Since the move to Bootstrap had not yet reached the Studio part of the
|
||||
edx-platform, most of the styling clashed with the Bootstrap CSS that
|
||||
studio-frontend components introduced. And, the Bootstrap styles were also
|
||||
leaking outside of the studio-frontend embedded component `div` and affecting
|
||||
the rest of the Studio page around it.
|
||||
|
||||
|
||||
|
||||
We were able to prevent styles leaking outside of the studio-frontend component
|
||||
by scoping all CSS to only the `div` that wraps the component. Thanks to the
|
||||
Webpack [postcss-loader](https://github.com/postcss/postcss-loader) and the
|
||||
[postcss-prepend-selector](https://github.com/ledniy/postcss-prepend-selector)
|
||||
we were able to automatically scope all of our CSS selectors to that `div` in
|
||||
our build process.
|
||||
|
||||
Preventing the Studio styles from affecting our studio-frontend component was a
|
||||
much harder problem because it means avoiding the inherently cascading nature of
|
||||
CSS. A common solution to this issue is to place the 3rd-party component inside
|
||||
of an
|
||||
[`iframe`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe)
|
||||
element, which essentially creates a completely separate sub-page where both CSS
|
||||
and JavaScript are isolated from the containing page. Because `iframe`s
|
||||
introduce many other performance and styling issues, we wanted to find a
|
||||
different solution to isolating CSS.
|
||||
|
||||
The CSS style [`all:
|
||||
initial`](https://developer.mozilla.org/en-US/docs/Web/CSS/all) allows
|
||||
resetting all properties on an element to their initial values as defined in the
|
||||
CSS spec. Placing this style under a wildcard selector in studio-frontend
|
||||
allowed us to reset all inherited props from the legacy Studio styles without
|
||||
having to enumerate them all by hand.
|
||||
|
||||
```css
|
||||
* {
|
||||
all: initial;
|
||||
}
|
||||
```
|
||||
|
||||
While this CSS property doesn’t have broad browser support yet, we were able to
|
||||
polyfill it thanks to postcss with the
|
||||
[postcss-initial](https://github.com/maximkoretskiy/postcss-initial) plugin.
|
||||
|
||||
However, this resets the styles to *nothing*. For example, all `div`s are
|
||||
displayed inline. To return the styles back to to some sane browser default we
|
||||
had to re-apply a browser default stylesheet. You can read more about this
|
||||
technique at
|
||||
[default-stylesheet](https://github.com/thallada/default-stylesheet).
|
||||
|
||||
From there, Bootstrap’s
|
||||
[reboot](https://getbootstrap.com/docs/4.0/content/reboot/) normalizes the
|
||||
browser-specific styling to a common baseline and then applies the Bootstrap
|
||||
styles conflict-free from the surrounding CSS cascade.
|
||||
|
||||
There's a candidate recommendation in CSS for a [`contains`
|
||||
property](https://www.w3.org/TR/css-contain-1/), which will "allow strong,
|
||||
predictable isolation of a subtree from the rest of the page". I hope that it
|
||||
will provide a much more elegant solution to this problem once browsers support
|
||||
it.
|
||||
|
||||
## Internationalization
|
||||
|
||||
Another major challenge with separating out the frontend from edx-platform was
|
||||
that most of our internationalization tooling was instrumented inside the
|
||||
edx-platform. So, in order to display text in studio-frontend components in the
|
||||
correct language we either had to pass already-translated strings from the
|
||||
edx-platform into studio-frontend, or set-up translations inside
|
||||
studio-frontend.
|
||||
|
||||
We opted for the latter because it kept the content close to the code that used
|
||||
it. Every display string in a component is stored in a
|
||||
[displayMessages.jsx](https://github.com/edx/studio-frontend/blob/master/src/components/AssetsTable/displayMessages.jsx)
|
||||
file and then imported and referenced by an id within the component. A periodic
|
||||
job extracts these strings from the project, pushes them up to our translations
|
||||
service [Transifex](https://www.transifex.com/), and pulls any new translations
|
||||
to store them in our NPM package.
|
||||
|
||||
Because Transifex’s `KEYVALUEJSON` file format does not allow for including
|
||||
comments in the strings for translation, [Eric](https://github.com/efischer19)
|
||||
created a library called [reactifex](https://github.com/efischer19/reactifex)
|
||||
that will send the comments in separate API calls.
|
||||
|
||||
Studio includes the user’s language in the context that it sends a
|
||||
studio-frontend component for initialization. Using this, the component can
|
||||
display the message for that language if it exists. If it does not, then it will
|
||||
display the original message in English and [wrap it in a `span` with `lang="en"`
|
||||
as an
|
||||
attribute](https://github.com/edx/studio-frontend/blob/master/src/utils/i18n/formattedMessageWrapper.jsx)
|
||||
so that screen-readers know to read it in English even if their default is some
|
||||
other language.
|
||||
|
||||
Read more about studio-frontend’s internationalization process in [the
|
||||
documentation that Eric
|
||||
wrote](https://github.com/edx/studio-frontend/blob/master/src/data/i18n/README.md).
|
||||
|
||||
## Developing with Docker
|
||||
|
||||
To normalize the development environment across the whole studio-frontend team,
|
||||
development is done in a Docker container. This is a minimal Ubuntu 16.04
|
||||
container with specific version of Node 8 installed and its only purpose is to
|
||||
run Webpack. This follows the pattern established in [OEP-5: Pre-built
|
||||
Development
|
||||
Environments](https://open-edx-proposals.readthedocs.io/en/latest/oep-0005-arch-containerize-devstack.html)
|
||||
for running a single Docker container per process that developers can easily
|
||||
start without installing dependencies.
|
||||
|
||||
Similar to edX’s [devstack](https://github.com/edx/devstack) there is a Makefile
|
||||
with commands to start and stop the docker container. The docker container then
|
||||
immediately runs [`npm run
|
||||
start`](https://github.com/edx/studio-frontend/blob/master/package.json#L12),
|
||||
which runs Webpack with the
|
||||
[webpack-dev-server](https://github.com/webpack/webpack-dev-server). The
|
||||
webpack-dev-server is a node server that serves assets built by Webpack.
|
||||
[Studio-frontend's Webpack
|
||||
config](https://github.com/edx/studio-frontend/blob/master/config/webpack.dev.config.js#L94)
|
||||
makes this server available to the developer's host machine
|
||||
at `http://localhost:18011`.
|
||||
|
||||
With [hot-reload](https://webpack.js.org/concepts/hot-module-replacement/)
|
||||
enabled, developers can now visit that URL in their browser, edit source files
|
||||
in studio-frontend, and then see changes reflected instantly in their browser
|
||||
once Webpack finishes its incremental rebuild.
|
||||
|
||||
However, many studio-frontend components need to be able to talk to the
|
||||
edx-platform Studio backend Django server. Using [docker’s network connect
|
||||
feature](https://docs.docker.com/compose/networking/#use-a-pre-existing-network)
|
||||
the studio-frontend container can join the developer’s existing docker devstack
|
||||
network so that the studio-frontend container can make requests to the docker
|
||||
devstack Studio container at `http://edx.devstack.studio:18010/` and Studio can
|
||||
access studio-frontend at `http://dahlia.studio-fronend:18011/`.
|
||||
|
||||
The webpack-dev-server can now [proxy all
|
||||
requests](https://github.com/edx/studio-frontend/blob/master/config/webpack.dev.config.js#L101)
|
||||
to Studio API endpoints (like `http://localhost:18011/assets`)
|
||||
to `http://edx.devstack.studio:18010/`.
|
||||
|
||||
## Developing within Docker Devstack Studio
|
||||
|
||||
Since studio-frontend components will be embedded inside of an existing Studio
|
||||
page shell, it’s often useful to develop on studio-frontend containers inside of
|
||||
this set-up. [This can be
|
||||
done](https://github.com/edx/studio-frontend#development-inside-devstack-studio)
|
||||
by setting a variable in the devstack's `cms/envs/private.py`:
|
||||
|
||||
```python
|
||||
STUDIO_FRONTEND_CONTAINER_URL = 'http://localhost:18011'
|
||||
```
|
||||
|
||||
This setting is checked in the Studio Mako templates wherever studio-frontend
|
||||
components are embedded. If it is set to a value other than `None`, then the
|
||||
templates will request assets from that URL instead of the Studio's own static
|
||||
assets directory. When a developer loads a Studio page with an embedded
|
||||
studio-frontend component, their studio-frontend webpack-dev-server will be
|
||||
requested at that URL. Similarly to developing on studio-frontend in isolation,
|
||||
edits to source files will trigger a Webpack compilation and the Studio page
|
||||
will be hot-reloaded or reloaded to reflect the changes automatically.
|
||||
|
||||
Since the studio-frontend JS loaded on `localhost:18010` is now requesting the
|
||||
webpack-dev-server on `localhost:18011`,
|
||||
an [`Access-Control-Allow-Origin` header](https://github.com/edx/studio-frontend/blob/master/config/webpack.dev.config.js#L98)
|
||||
has to be configured on the webpack-dev-server to get around CORS violations.
|
||||
|
||||
|
||||
|
||||
## Deploying to Production
|
||||
|
||||
[Each release of
|
||||
studio-frontend](https://github.com/edx/studio-frontend#releases) will upload
|
||||
the `/dist` files built by Webpack in production mode to
|
||||
[NPM](https://www.npmjs.com/package/@edx/studio-frontend). edx-platform
|
||||
requires a particular version of studio-frontend in its
|
||||
[`package.json`](https://github.com/edx/edx-platform/blob/master/package.json#L7).
|
||||
When a new release of edx-platform is made, `paver update_assets` will run
|
||||
which will copy all of the files in the
|
||||
`node_modules/@edx/studio-frontend/dist/` to the Studio static folder.
|
||||
Because `STUDIO_FRONTEND_CONTAINER_URL` will be `None` in production, it will be
|
||||
ignored, and Studio pages will request studio-frontend assets from that static
|
||||
folder.
|
||||
|
||||
## Future
|
||||
|
||||
Instead of “bringing the new into the old”, we’d eventually like to move to a
|
||||
model where we “work in the new and bring in the old if necessary”. We could
|
||||
host studio-frontend statically on a completely separate server which talks to
|
||||
Studio via a REST (or [GraphQL](https://graphql.org/)) API. This approach would
|
||||
eliminate the complexity around CSS isolation and bring big performance wins for
|
||||
our users, but it would require us to rewrite more of Studio.
|
||||
@ -1,758 +0,0 @@
|
||||
---
|
||||
title: "Generating icosahedrons and hexspheres in Rust"
|
||||
layout: post
|
||||
image: /img/blog/hexsphere_colored_7.png
|
||||
---
|
||||
|
||||
I've been trying to learn [Rust](https://www.rust-lang.org/) lately, the hot new
|
||||
systems programming language. One of the projects I wanted to tackle with the
|
||||
speed of Rust was generating 3D polyhedron shapes. Specifically, I wanted to
|
||||
implement something like the [Three.js
|
||||
`IcosahedronGeometry`](https://threejs.org/docs/#api/en/geometries/IcosahedronGeometry)
|
||||
in Rust. If you try to generate
|
||||
[icosahedron](https://en.wikipedia.org/wiki/Icosahedron)s in Three.js over any
|
||||
detail level over 5 the whole browser will slow to a crawl. I think we can do
|
||||
better in Rust!
|
||||
|
||||
Furthermore, I wanted to generate a hexsphere: a sphere composed of hexagon
|
||||
faces and 12 pentagon faces, otherwise known as a truncated icosahedron or the
|
||||
[Goldberg polyhedron](https://en.wikipedia.org/wiki/Goldberg_polyhedron). The
|
||||
shape would be ideal for a game since (almost) every tile would have the same
|
||||
area and six sides to defend or attack from. There's a few [Javascript projects
|
||||
for generating hexspheres](https://www.robscanlon.com/hexasphere/). Most of them
|
||||
generate the shape by starting with a subdivided icosahedron and then truncating
|
||||
the sides into hexagons. Though, there [exist other methods for generating the
|
||||
hexsphere
|
||||
shape](https://stackoverflow.com/questions/46777626/mathematically-producing-sphere-shaped-hexagonal-grid).
|
||||
|
||||
**Play around with all of these shapes in your browser at:
|
||||
[https://www.hallada.net/planet/](https://www.hallada.net/planet/).**
|
||||
|
||||
So, how would we go about generating a hexsphere from scratch?
|
||||
|
||||
<!--excerpt-->
|
||||
|
||||
### The Icosahedron Seed
|
||||
|
||||
To start our sculpture, we need our ball of clay. The most basic shape that we
|
||||
start with can be defined by its 20 triangle faces and 12 vertices: the regular
|
||||
icosahedron. If you've ever played Dungeons and Dragons, this is the 20-sided
|
||||
die.
|
||||
|
||||
To define this basic shape in Rust, we first need to define a few structs. The
|
||||
most basic unit we need is a 3D vector which describes a single point in 3D
|
||||
space with a X, Y, and Z float values. I could have defined this myself, but to
|
||||
avoid having to implement a bunch of vector operations (like add, subtract,
|
||||
multiply, etc.) I chose to import
|
||||
[`Vector3`](https://docs.rs/cgmath/0.17.0/cgmath/struct.Vector3.html) from the
|
||||
[cgmath crate](https://crates.io/crates/cgmath).
|
||||
|
||||
The next struct we need is `Triangle`. This will define a face between three
|
||||
vertices:
|
||||
|
||||
```rust
|
||||
#[derive(Debug)]
|
||||
pub struct Triangle {
|
||||
pub a: usize,
|
||||
pub b: usize,
|
||||
pub c: usize,
|
||||
}
|
||||
|
||||
impl Triangle {
|
||||
fn new(a: usize, b: usize, c: usize) -> Triangle {
|
||||
Triangle { a, b, c }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
We use `usize` for the three points of the triangle because they are indices
|
||||
into a [`Vec`](https://doc.rust-lang.org/std/vec/struct.Vec.html) of `Vector3`s.
|
||||
|
||||
To keep these all together, I'll define a `Polyhedron` struct:
|
||||
|
||||
```rust
|
||||
#[derive(Debug)]
|
||||
pub struct Polyhedron {
|
||||
pub positions: Vec<Vector3>,
|
||||
pub cells: Vec<Triangle>,
|
||||
}
|
||||
```
|
||||
|
||||
With this, we can define the regular icosahedron:
|
||||
|
||||
```rust
|
||||
impl Polyhedron {
|
||||
pub fn regular_isocahedron() -> Polyhedron {
|
||||
let t = (1.0 + (5.0 as f32).sqrt()) / 2.0;
|
||||
Polyhedron {
|
||||
positions: vec![
|
||||
Vector3::new(-1.0, t, 0.0),
|
||||
Vector3::new(1.0, t, 0.0),
|
||||
Vector3::new(-1.0, -t, 0.0),
|
||||
Vector3::new(1.0, -t, 0.0),
|
||||
Vector3::new(0.0, -1.0, t),
|
||||
Vector3::new(0.0, 1.0, t),
|
||||
Vector3::new(0.0, -1.0, -t),
|
||||
Vector3::new(0.0, 1.0, -t),
|
||||
Vector3::new(t, 0.0, -1.0),
|
||||
Vector3::new(t, 0.0, 1.0),
|
||||
Vector3::new(-t, 0.0, -1.0),
|
||||
Vector3::new(-t, 0.0, 1.0),
|
||||
],
|
||||
cells: vec![
|
||||
Triangle::new(0, 11, 5),
|
||||
Triangle::new(0, 5, 1),
|
||||
Triangle::new(0, 1, 7),
|
||||
Triangle::new(0, 7, 10),
|
||||
Triangle::new(0, 10, 11),
|
||||
Triangle::new(1, 5, 9),
|
||||
Triangle::new(5, 11, 4),
|
||||
Triangle::new(11, 10, 2),
|
||||
Triangle::new(10, 7, 6),
|
||||
Triangle::new(7, 1, 8),
|
||||
Triangle::new(3, 9, 4),
|
||||
Triangle::new(3, 4, 2),
|
||||
Triangle::new(3, 2, 6),
|
||||
Triangle::new(3, 6, 8),
|
||||
Triangle::new(3, 8, 9),
|
||||
Triangle::new(4, 9, 5),
|
||||
Triangle::new(2, 4, 11),
|
||||
Triangle::new(6, 2, 10),
|
||||
Triangle::new(8, 6, 7),
|
||||
Triangle::new(9, 8, 1),
|
||||
],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### JSON Serialization
|
||||
|
||||
To prove this works, we need to be able to output our shape to some format that
|
||||
will be able to be rendered. Coming from a JS background, I'm only familiar with
|
||||
rendering shapes with WebGL. So, I need to be able to serialize the shape to
|
||||
JSON so I can load it in JS.
|
||||
|
||||
There's an amazing library in Rust called
|
||||
[serde](https://crates.io/crates/serde) that will make this very
|
||||
straightforward. We just need to import it and `impl Serialize` for all of our
|
||||
structs.
|
||||
|
||||
The JSON structure we want will look like this. This is what Three.js expects
|
||||
when initializing
|
||||
[`BufferGeometry`](https://threejs.org/docs/#api/en/core/BufferGeometry).
|
||||
|
||||
```json
|
||||
{
|
||||
"positions": [
|
||||
[
|
||||
-0.8506508,
|
||||
0,
|
||||
0.5257311
|
||||
],
|
||||
...
|
||||
],
|
||||
"cells": [
|
||||
[
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
],
|
||||
...
|
||||
],
|
||||
}
|
||||
```
|
||||
|
||||
For the `"cells"` array, we'll need to serialize `Triangle` into an array of 3
|
||||
integer arrays:
|
||||
|
||||
```rust
|
||||
impl Serialize for Triangle {
|
||||
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
|
||||
where
|
||||
S: Serializer,
|
||||
{
|
||||
let vec_indices = vec![self.a, self.b, self.c];
|
||||
let mut seq = serializer.serialize_seq(Some(vec_indices.len()))?;
|
||||
for index in vec_indices {
|
||||
seq.serialize_element(&index)?;
|
||||
}
|
||||
seq.end()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
I had some trouble serializing the `cgmath::Vector3` to an array, so I made my
|
||||
own type that wrapped `Vector3` that could be serialized to an array of 3
|
||||
floats.
|
||||
|
||||
```rust
|
||||
#[derive(Debug)]
|
||||
pub struct ArraySerializedVector(pub Vector3<f32>);
|
||||
|
||||
impl Serialize for ArraySerializedVector {
|
||||
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
|
||||
where
|
||||
S: Serializer,
|
||||
{
|
||||
let values = vec![self.0.x, self.0.y, self.0.z];
|
||||
let mut seq = serializer.serialize_seq(Some(values.len()))?;
|
||||
for value in values {
|
||||
seq.serialize_element(&value)?;
|
||||
}
|
||||
seq.end()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
And now `Polyhedron` needs to use this new type and implement `Serialize` for
|
||||
the whole shape to get serialized:
|
||||
|
||||
```rust
|
||||
#[derive(Serialize, Debug)]
|
||||
pub struct Polyhedron {
|
||||
pub positions: Vec<ArraySerializedVector>,
|
||||
pub cells: Vec<Triangle>,
|
||||
}
|
||||
```
|
||||
|
||||
The actual serialization is done with:
|
||||
|
||||
```rust
|
||||
fn write_to_json_file(polyhedron: Polyhedron, path: &Path) {
|
||||
let mut json_file = File::create(path).expect("Can't create file");
|
||||
let json = serde_json::to_string(&polyhedron).expect("Problem serializing");
|
||||
json_file
|
||||
.write_all(json.as_bytes())
|
||||
.expect("Can't write to file");
|
||||
}
|
||||
```
|
||||
|
||||
On the JS side, the `.json` file can be read and simply fed into either Three.js
|
||||
or [regl](https://github.com/regl-project/reg) to be rendered in WebGL ([more on
|
||||
that later](#rendering-in-webgl-with-regl)).
|
||||
|
||||
|
||||
|
||||
## Subdivided Icosahedron
|
||||
|
||||
Now, we need to take our regular icosahedron and subdivide its faces N number of
|
||||
times to generate an icosahedron with a detail level of N.
|
||||
|
||||
I pretty much copied must of [the subdividing code from
|
||||
Three.js](https://github.com/mrdoob/three.js/blob/34dc2478c684066257e4e39351731a93c6107ef5/src/geometries/PolyhedronGeometry.js#L90)
|
||||
directly into Rust.
|
||||
|
||||
I won't bore you with the details here, you can find the function
|
||||
[here](https://github.com/thallada/icosahedron/blob/9643757df245e29f5ecfbb25f9a2c06b3a4e1217/src/lib.rs#L160-L205).
|
||||
|
||||
|
||||
|
||||
### Truncated Icosahedron
|
||||
|
||||
Now we get to the meat of this project. Transforming an icosahedron into a
|
||||
hexsphere by
|
||||
[truncating](https://en.wikipedia.org/wiki/Truncation_%28geometry%29) the points
|
||||
of the icosahedron into hexagon and pentagon faces.
|
||||
|
||||
You can imagine this operation as literally cutting off the points of the
|
||||
subdivided icosahedron at exactly the midpoint between the point and it's six or
|
||||
five neighboring points.
|
||||
|
||||
|
||||
([image source](http://www.oz.nthu.edu.tw/~u9662122/DualityProperty.html))
|
||||
|
||||
In this image you can see the regular icosahedron (0 subdivisions) in wireframe
|
||||
with a yellow shape underneath which is the result of all 12 points truncated to
|
||||
12 pentagon faces, in other words: the [regular
|
||||
dodecahedron](https://en.wikipedia.org/wiki/Dodecahedron).
|
||||
|
||||
You can see that the points of the new pentagon faces will be the exact center
|
||||
of the original triangular faces. It should now make sense why truncating a
|
||||
shape with 20 faces of 3 edges each results in a shape with 12 faces of 5 edges
|
||||
each. Each pair multiplied still equals 60.
|
||||
|
||||
#### Algorithm
|
||||
|
||||
There are many different algorithms you could use to generate the truncated
|
||||
shape, but this is roughly what I came up with:
|
||||
|
||||
1. Store a map of every icosahedron vertex to faces composed from that vertex
|
||||
(`vert_to_faces`).
|
||||
|
||||
2. Calculate and cache the [centroid](https://en.wikipedia.org/wiki/Centroid) of
|
||||
every triangle in the icosahedron (`triangle_centroids`).
|
||||
|
||||
3. For every vertex in the original icosahedron:
|
||||
|
||||
4. Find the center point between all of centroids of all of the faces for that
|
||||
vertex (`center_point`). This is essentially the original icosahedron point
|
||||
but lowered towards the center of the polygon since it will eventually be the
|
||||
center of a new flat hexagon face.
|
||||
|
||||
|
||||
|
||||
5. For every triangle face composed from the original vertex:
|
||||
|
||||
|
||||
|
||||
6. Sort the vertices of the triangle face so there is a vertex `A` in the center
|
||||
of the fan like in the image, and two other vertices `B` and `C` at the edges
|
||||
of the hexagon.
|
||||
|
||||
7. Find the centroid of the selected face. This will be one of the five or six
|
||||
points of the new pentagon or hexagon (in brown in diagram below:
|
||||
`triangleCentroid`).
|
||||
|
||||
8. Find the mid point between `AB` and `AC` (points `midAB` and `midAC` in
|
||||
diagram).
|
||||
|
||||
9. With these mid points and the face centroid, we now have two new triangles
|
||||
(in orange below) that form one-fifth or one-sixth of the final pentagon or
|
||||
hexagon face. Add the points of the triangle to the `positions` array. Add
|
||||
the two new triangles composed from those vertices as indexes into the
|
||||
`positions` array to the `cells` array. We need to compose the pentagon or
|
||||
hexagon out of triangles because in graphics everything is a triangle, and
|
||||
this is the simplest way to tile either shape with triangles:
|
||||
|
||||
|
||||
|
||||
10. Go to step 5 until all faces of the icosahedron vertex have been visited.
|
||||
Save indices to all new triangles in the `cells` array, which now form a
|
||||
complete pentagon or hexagon face, to the `faces` array.
|
||||
|
||||
|
||||
|
||||
11. Go to step 3 until all vertices in the icosahedron have been visited. The
|
||||
truncated icosahedron is now complete.
|
||||
|
||||
|
||||
|
||||
#### Code
|
||||
|
||||
The `truncate` function calls out to a bunch of other functions, so [here's a
|
||||
link to the function within the context of the whole
|
||||
file](https://github.com/thallada/icosahedron/blob/9643757df245e29f5ecfbb25f9a2c06b3a4e1217/src/lib.rs#L227).
|
||||
|
||||
### Calculating Normals
|
||||
|
||||
It took me a surprisingly long time to figure out how to compute
|
||||
[normals](https://en.wikipedia.org/wiki/Normal_(geometry)) for the truncated
|
||||
icosahedron. I tried just using an out-of-the-box solution like
|
||||
[angle-normals](https://github.com/mikolalysenko/angle-normals/blob/master/angle-normals.js)
|
||||
which could supposedly calculate the normal vectors for you, but they came out
|
||||
all wrong.
|
||||
|
||||
|
||||
|
||||
So, I tried doing it myself. Most tutorials on computing normal vectors for a
|
||||
mesh assume that it is tiled in a particular way. But, my algorithm spins around
|
||||
icosahedron points in all different directions, and so the triangle points are
|
||||
not uniformly in clockwise or counter-clockwise order.
|
||||
|
||||
I could have sorted these points into the correct order, but I found it easier
|
||||
to instead just detect when the normal was pointing the wrong way and just
|
||||
invert it.
|
||||
|
||||
```rust
|
||||
pub fn compute_triangle_normals(&mut self) {
|
||||
let origin = Vector3::new(0.0, 0.0, 0.0);
|
||||
for i in 0..self.cells.len() {
|
||||
let vertex_a = &self.positions[self.cells[i].a].0;
|
||||
let vertex_b = &self.positions[self.cells[i].b].0;
|
||||
let vertex_c = &self.positions[self.cells[i].c].0;
|
||||
|
||||
let e1 = vertex_a - vertex_b;
|
||||
let e2 = vertex_c - vertex_b;
|
||||
let mut no = e1.cross(e2);
|
||||
|
||||
// detect and correct inverted normal
|
||||
let dist = vertex_b - origin;
|
||||
if no.dot(dist) < 0.0 {
|
||||
no *= -1.0;
|
||||
}
|
||||
|
||||
let normal_a = self.normals[self.cells[i].a].0 + no;
|
||||
let normal_b = self.normals[self.cells[i].b].0 + no;
|
||||
let normal_c = self.normals[self.cells[i].c].0 + no;
|
||||
|
||||
self.normals[self.cells[i].a] = ArraySerializedVector(normal_a);
|
||||
self.normals[self.cells[i].b] = ArraySerializedVector(normal_b);
|
||||
self.normals[self.cells[i].c] = ArraySerializedVector(normal_c);
|
||||
}
|
||||
|
||||
for normal in self.normals.iter_mut() {
|
||||
*normal = ArraySerializedVector(normal.0.normalize());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Assigning Random Face Colors
|
||||
|
||||
Finally, all that's left to generate is the face colors. The only way I could
|
||||
figure out how to individually color a shape's faces in WebGL was to pass a
|
||||
color per vertex. The issue with this is that each vertex of the generated
|
||||
shapes could be shared between many different faces.
|
||||
|
||||
How can we solve this? At the cost of memory, we can just duplicate a vertex
|
||||
every time it's used by a different triangle. That way no vertex is shared.
|
||||
|
||||
This can be done after a shape has been generated with shared vertices.
|
||||
|
||||
```rust
|
||||
pub fn unique_vertices(&mut self, other: Polyhedron) {
|
||||
for triangle in other.cells {
|
||||
let vertex_a = other.positions[triangle.a].0;
|
||||
let vertex_b = other.positions[triangle.b].0;
|
||||
let vertex_c = other.positions[triangle.c].0;
|
||||
let normal_a = other.normals[triangle.a].0;
|
||||
let normal_b = other.normals[triangle.b].0;
|
||||
let normal_c = other.normals[triangle.c].0;
|
||||
|
||||
self.positions.push(ArraySerializedVector(vertex_a));
|
||||
self.positions.push(ArraySerializedVector(vertex_b));
|
||||
self.positions.push(ArraySerializedVector(vertex_c));
|
||||
self.normals.push(ArraySerializedVector(normal_a));
|
||||
self.normals.push(ArraySerializedVector(normal_b));
|
||||
self.normals.push(ArraySerializedVector(normal_c));
|
||||
self.colors
|
||||
.push(ArraySerializedVector(Vector3::new(1.0, 1.0, 1.0)));
|
||||
self.colors
|
||||
.push(ArraySerializedVector(Vector3::new(1.0, 1.0, 1.0)));
|
||||
self.colors
|
||||
.push(ArraySerializedVector(Vector3::new(1.0, 1.0, 1.0)));
|
||||
let added_index = self.positions.len() - 1;
|
||||
self.cells
|
||||
.push(Triangle::new(added_index - 2, added_index - 1, added_index));
|
||||
}
|
||||
self.faces = other.faces;
|
||||
}
|
||||
```
|
||||
|
||||
With unique vertices, we can now generate a random color per face with the [rand
|
||||
crate](https://crates.io/crates/rand).
|
||||
|
||||
|
||||
```rust
|
||||
pub fn assign_random_face_colors(&mut self) {
|
||||
let mut rng = rand::thread_rng();
|
||||
for i in 0..self.faces.len() {
|
||||
let face_color = Vector3::new(rng.gen(), rng.gen(), rng.gen());
|
||||
|
||||
for c in 0..self.faces[i].len() {
|
||||
let face_cell = &self.cells[self.faces[i][c]];
|
||||
|
||||
self.colors[face_cell.a] = ArraySerializedVector(face_color);
|
||||
self.colors[face_cell.b] = ArraySerializedVector(face_color);
|
||||
self.colors[face_cell.c] = ArraySerializedVector(face_color);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Binary Serialization
|
||||
|
||||
Now that we have to duplicate vertices for individual face colors, the size of
|
||||
our JSON outputs are getting quite big:
|
||||
|
||||
| File | Size |
|
||||
|---|---|
|
||||
| icosahedron_r1_d6.json | 28 MB |
|
||||
| icosahedron_r1_d7.json | 113 MB |
|
||||
| hexsphere_r1_d5.json | 42 MB |
|
||||
| hexsphere_r1_d6.json | 169 MB |
|
||||
|
||||
Since all of our data is just floating point numbers, we could reduce the size
|
||||
of the output considerably by using a binary format instead.
|
||||
|
||||
I used the [byteorder](https://docs.rs/byteorder/1.3.2/byteorder/) crate to
|
||||
write out all of the `Vec`s in my `Polyhedron` struct to a binary file in
|
||||
little-endian order.
|
||||
|
||||
The binary format is laid out as:
|
||||
|
||||
1. 1 32 bit unsigned integer specifying the number of vertices (`V`)
|
||||
2. 1 32 bit unsigned integer specifying the number of triangles (`T`)
|
||||
3. `V` * 3 number of 32 bit floats for every vertex's x, y, and z coordinate
|
||||
4. `V` * 3 number of 32 bit floats for the normals of every vertex
|
||||
5. `V` * 3 number of 32 bit floats for the color of every vertex
|
||||
6. `T` * 3 number of 32 bit unsigned integers for the 3 indices into the vertex
|
||||
array that make every triangle
|
||||
|
||||
The `write_to_binary_file` function which does all that is
|
||||
[here](https://github.com/thallada/icosahedron/blob/9643757df245e29f5ecfbb25f9a2c06b3a4e1217/src/bin.rs#L13).
|
||||
|
||||
That's a lot better:
|
||||
|
||||
| File | Size |
|
||||
|---|---|
|
||||
| icosahedron_r1_d6.bin | 9.8 MB |
|
||||
| icosahedron_r1_d7.bin | 11 MB |
|
||||
| hexsphere_r1_d5.bin | 14 MB |
|
||||
| hexsphere_r1_d6.bin | 58 MB |
|
||||
|
||||
On the JavaScript side, the binary files can be read into `Float32Array`s like
|
||||
this:
|
||||
|
||||
```javascript
|
||||
fetch(binaryFile)
|
||||
.then(response => response.arrayBuffer())
|
||||
.then(buffer => {
|
||||
let reader = new DataView(buffer);
|
||||
let numVertices = reader.getUint32(0, true);
|
||||
let numCells = reader.getUint32(4, true);
|
||||
let shape = {
|
||||
positions: new Float32Array(buffer, 8, numVertices * 3),
|
||||
normals: new Float32Array(buffer, numVertices * 12 + 8, numVertices * 3),
|
||||
colors: new Float32Array(buffer, numVertices * 24 + 8, numVertices * 3),
|
||||
cells: new Uint32Array(buffer, numVertices * 36 + 8, numCells * 3),
|
||||
})
|
||||
```
|
||||
|
||||
### Rendering in WebGL with Regl
|
||||
|
||||
I was initially rendering the shapes with Three.js but switched to
|
||||
[regl](https://github.com/regl-project/regl) because it seemed like a more
|
||||
direct abstraction over WebGL. It makes setting up a WebGL renderer incredibly
|
||||
easy compared to all of the dozens cryptic function calls you'd have to
|
||||
otherwise use.
|
||||
|
||||
This is pretty much all of the rendering code using regl in my [3D hexsphere and
|
||||
icosahedron viewer project](https://github.com/thallada/planet).
|
||||
|
||||
```javascript
|
||||
const drawShape = hexsphere => regl({
|
||||
vert: `
|
||||
precision mediump float;
|
||||
uniform mat4 projection, view;
|
||||
attribute vec3 position, normal, color;
|
||||
varying vec3 fragNormal, fragPosition, fragColor;
|
||||
void main() {
|
||||
fragNormal = normal;
|
||||
fragPosition = position;
|
||||
fragColor = color;
|
||||
gl_Position = projection * view * vec4(position, 1.0);
|
||||
}`,
|
||||
|
||||
frag: `
|
||||
precision mediump float;
|
||||
struct Light {
|
||||
vec3 color;
|
||||
vec3 position;
|
||||
};
|
||||
uniform Light lights[1];
|
||||
varying vec3 fragNormal, fragPosition, fragColor;
|
||||
void main() {
|
||||
vec3 normal = normalize(fragNormal);
|
||||
vec3 light = vec3(0.1, 0.1, 0.1);
|
||||
for (int i = 0; i < 1; i++) {
|
||||
vec3 lightDir = normalize(lights[i].position - fragPosition);
|
||||
float diffuse = max(0.0, dot(lightDir, normal));
|
||||
light += diffuse * lights[i].color;
|
||||
}
|
||||
gl_FragColor = vec4(fragColor * light, 1.0);
|
||||
}`,
|
||||
|
||||
attributes: {
|
||||
position: hexsphere.positions,
|
||||
normal: hexsphere.normals,
|
||||
color: hexsphere.colors,
|
||||
},
|
||||
elements: hexsphere.cells,
|
||||
uniforms: {
|
||||
"lights[0].color": [1, 1, 1],
|
||||
"lights[0].position": ({ tick }) => {
|
||||
const t = 0.008 * tick
|
||||
return [
|
||||
1000 * Math.cos(t),
|
||||
1000 * Math.sin(t),
|
||||
1000 * Math.sin(t)
|
||||
]
|
||||
},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
I also imported [regl-camera](https://github.com/regl-project/regl-camera) which
|
||||
handled all of the complex viewport code for me.
|
||||
|
||||
It was fairly easy to get a simple renderer working quickly in regl, but I
|
||||
couldn't find many examples of more complex projects using regl. Unfortunately,
|
||||
the project looks a bit unmaintained these days as well. If I'm going to
|
||||
continue with rendering in WebGL, I think I will try out
|
||||
[Babylon.js](https://www.babylonjs.com/) instead.
|
||||
|
||||
### Running in WebAssembly
|
||||
|
||||
Since rust can be compiled down to wasm and then run in the browser, I briefly
|
||||
tried getting the project to run completely in the browser.
|
||||
|
||||
The [wasm-pack](https://github.com/rustwasm/wasm-pack) tool made it pretty easy
|
||||
to get started. My main struggle was figuring out an efficient way to get the
|
||||
megabytes of generated shape data into the JavaScript context so it could be
|
||||
rendered in WebGL.
|
||||
|
||||
The best I could come up with was to export all of my structs into flat
|
||||
`Vec<f32>`s and then create `Float32Array`s from the JS side that are views into
|
||||
wasm's memory.
|
||||
|
||||
To export:
|
||||
|
||||
```rust
|
||||
pub fn fill_exports(&mut self) {
|
||||
for position in &self.positions {
|
||||
self.export_positions.push(position.0.x);
|
||||
self.export_positions.push(position.0.y);
|
||||
self.export_positions.push(position.0.z);
|
||||
}
|
||||
for normal in &self.normals {
|
||||
self.export_normals.push(normal.0.x);
|
||||
self.export_normals.push(normal.0.y);
|
||||
self.export_normals.push(normal.0.z);
|
||||
}
|
||||
for color in &self.colors {
|
||||
self.export_colors.push(color.0.x);
|
||||
self.export_colors.push(color.0.y);
|
||||
self.export_colors.push(color.0.z);
|
||||
}
|
||||
for cell in &self.cells {
|
||||
self.export_cells.push(cell.a as u32);
|
||||
self.export_cells.push(cell.b as u32);
|
||||
self.export_cells.push(cell.c as u32);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
And then the wasm `lib.rs`:
|
||||
|
||||
```rust
|
||||
use byteorder::{LittleEndian, WriteBytesExt};
|
||||
use js_sys::{Array, Float32Array, Uint32Array};
|
||||
use wasm_bindgen::prelude::*;
|
||||
use web_sys::console;
|
||||
|
||||
mod icosahedron;
|
||||
|
||||
#[cfg(feature = "wee_alloc")]
|
||||
#[global_allocator]
|
||||
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;
|
||||
|
||||
#[wasm_bindgen(start)]
|
||||
pub fn main_js() -> Result<(), JsValue> {
|
||||
#[cfg(debug_assertions)]
|
||||
console_error_panic_hook::set_once();
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[wasm_bindgen]
|
||||
pub struct Hexsphere {
|
||||
positions: Float32Array,
|
||||
normals: Float32Array,
|
||||
colors: Float32Array,
|
||||
cells: Uint32Array,
|
||||
}
|
||||
|
||||
#[wasm_bindgen]
|
||||
pub fn shape_data() -> Result<Array, JsValue> {
|
||||
let radius = 1.0;
|
||||
let detail = 7;
|
||||
let mut hexsphere = icosahedron::Polyhedron::new_truncated_isocahedron(radius, detail);
|
||||
hexsphere.compute_triangle_normals();
|
||||
let mut unique_hexsphere = icosahedron::Polyhedron::new();
|
||||
unique_hexsphere.unique_vertices(hexsphere);
|
||||
unique_hexsphere.assign_random_face_colors();
|
||||
unique_hexsphere.fill_exports();
|
||||
|
||||
let positions = unsafe { Float32Array::view(&unique_hexsphere.export_positions) };
|
||||
let normals = unsafe { Float32Array::view(&unique_hexsphere.export_normals) };
|
||||
let colors = unsafe { Float32Array::view(&unique_hexsphere.export_colors) };
|
||||
let cells = unsafe { Uint32Array::view(&unique_hexsphere.export_cells) };
|
||||
|
||||
Ok(Array::of4(&positions, &normals, &colors, &cells))
|
||||
}
|
||||
```
|
||||
|
||||
With wasm-pack, I could import the wasm package, run the `shape_data()`
|
||||
function, and then read the contents as any other normal JS array.
|
||||
|
||||
```javascript
|
||||
let rust = import("../pkg/index.js")
|
||||
rust.then(module => {
|
||||
const shapeData = module.shape_data()
|
||||
const shape = {
|
||||
positions: shapeData[0],
|
||||
normals: shapeData[1],
|
||||
colors: shapeData[2],
|
||||
cells: shapeData[3],
|
||||
}
|
||||
...
|
||||
})
|
||||
```
|
||||
|
||||
I could side-step the issue of transferring data from Rust to JavaScript
|
||||
entirely by programming literally everything in WebAssembly. But the bindings
|
||||
from rust wasm to the WebGL API are still way too complicated compared to just
|
||||
using regl. Plus, I'd have to implement my own camera from scratch.
|
||||
|
||||
### The Stats
|
||||
|
||||
So how much faster is Rust than JavaScript in generating icosahedrons and
|
||||
hexspheres?
|
||||
|
||||
Here's how long it took with generating shapes in JS with Three.js in Firefox
|
||||
versus in native Rust with a i5-2500K 3.3 GHz CPU.
|
||||
|
||||
| Shape | JS generate time | Rust generate time |
|
||||
|---|---|---|
|
||||
| Icosahedron detail 6 | 768 ms | 28.23 ms |
|
||||
| Icosahedron detail 7 | 4.25 s | 128.81 ms |
|
||||
| Hexsphere detail 6 | 11.37 s | 403.10 ms |
|
||||
| Hexsphere detail 7 | 25.49 s | 1.85 s |
|
||||
|
||||
So much faster!
|
||||
|
||||
|
||||
### Todo
|
||||
|
||||
* Add a process that alters the shape post-generation. Part of the reason why I
|
||||
decided to fan the hexagon faces with so many triangles is that it also allows
|
||||
me to control the height of the faces better. This could eventually allow me
|
||||
to create mountain ranges and river valleys on a hexsphere planet. Stretching
|
||||
and pulling the edges of the polygon faces in random directions could add
|
||||
variation and make for a more organic looking hexsphere.
|
||||
|
||||
* Conversely, it would be nice to be able to run a process post-generation that
|
||||
could reduce the number of triangles by tiling the hexagons more efficiently
|
||||
when face elevation isn't needed.
|
||||
|
||||
* Add parameters to the generation that allows generating sections of the
|
||||
hexsphere / icosahedron. This will be essential for rendering very detailed
|
||||
polyhedrons since at a certain detail level it becomes impossible to render
|
||||
the entire shape at once.
|
||||
|
||||
In WebGL, figure out what part of the shape is in the current viewport and
|
||||
pass these parameters to the generation.
|
||||
|
||||
* Render the shapes in a native Rust graphics library instead of WebGL. I'm
|
||||
curious how much slower WebGL is making things.
|
||||
|
||||
* Parallelize the generation. Right now the generation is very CPU bound and
|
||||
each subdivide/truncate iteration is mostly independent from each other, so I
|
||||
think I could get some decent speed-up by allowing the process to run on
|
||||
multiple cores. Perhaps the [rayon](https://github.com/rayon-rs/rayon) crate
|
||||
could make this pretty straightforward.
|
||||
|
||||
* Find some way to avoid unique vertices. The size of the shape is *much* bigger
|
||||
because of this. There might be a way to keep shared vertices while also
|
||||
having a separate color per face by using texture mapping.
|
||||
|
||||
* In the renderer, implement face selection (point and click face and show an
|
||||
outline around selected face).
|
||||
|
||||
* In the renderer, implement fly-to-face zooming: given a face, fly the camera
|
||||
around the sphere in an orbit and then zoom in on the face.
|
||||
@ -1,686 +0,0 @@
|
||||
---
|
||||
title: "Modmapper: Putting every Skyrim mod on a map with Rust"
|
||||
layout: post
|
||||
image: /img/blog/modmapper.jpg
|
||||
---
|
||||
|
||||
[Modmapper](https://modmapper.com) is a website that I made that puts every mod
|
||||
for the game [Elder Scrolls V:
|
||||
Skyrim](https://en.wikipedia.org/wiki/The_Elder_Scrolls_V:_Skyrim) uploaded to
|
||||
[Nexus Mods](https://www.nexusmods.com/) on an interactive map.
|
||||
|
||||
<a href="https://modmapper.com" target="_blank">
|
||||
|
||||
</a>
|
||||
|
||||
You can view the map at [https://modmapper.com](https://modmapper.com).
|
||||
|
||||
Released in 2011, Skyrim is over a decade old now. But, its vast modding
|
||||
community has kept it alive and relevant to this day. [Skyrim is still in the
|
||||
top 50 games being played on Steam in 2022](https://steamcharts.com/top/p.2) and
|
||||
I think it's no coincidence that [it's also one of the most modded games
|
||||
ever](https://www.nexusmods.com/games?).
|
||||
|
||||
<!--excerpt-->
|
||||
|
||||
The enormous and enduring modding community around the Elder Scrolls games is
|
||||
why I have a special fondness for the series. I was 13 when I first got
|
||||
interested in programming through [making mods for Elder Scrolls IV:
|
||||
Oblivion](https://www.nexusmods.com/users/512579?tab=user+files&BH=2). I quickly
|
||||
realized I got way more satisfaction out of modding the game than actually
|
||||
playing it. I was addicted to being able to create whatever my mind imagined in
|
||||
my favorite game.
|
||||
|
||||
I was working on mod for Skyrim earlier in the year[^bazaarrealm] and was
|
||||
looking for the best places to put new buildings in the game world. I really
|
||||
wanted areas of the game world off the beaten (heavily-modded) path. After over
|
||||
a decade of modifications, there could be conflicts with hundreds of mods in any
|
||||
area I chose which could cause issues like multiple buildings overlapping or
|
||||
terrain changes causing floating rocks and trees.
|
||||
|
||||
<p>
|
||||
<div class="row">
|
||||
<figure>
|
||||
<img alt="Example of a conflict between two mods that both chose the
|
||||
same spot to put a lamp post and sign post so they are clipping"
|
||||
src="/img/blog/modmapper-clipping-example2.jpg" />
|
||||
<figurecaption>
|
||||
<em>
|
||||
Example of a conflict between two mods that both chose the same
|
||||
spot to put a lamp post and sign post so they are clipping.
|
||||
Screenshot by <a
|
||||
href="https://www.nexusmods.com/users/63732336">
|
||||
AndreySG</a>.
|
||||
</em>
|
||||
</figurecaption>
|
||||
</figure>
|
||||
<figure>
|
||||
<img alt="Example of a conflict between two mods that both chose the
|
||||
same spot to put a building and rock so they are clipping"
|
||||
src="/img/blog/modmapper-clipping-example1.jpg" />
|
||||
<figurecaption>
|
||||
<em>
|
||||
Conflict between a building and rock. Screenshot by <a
|
||||
href="https://www.reddit.com/user/LewdManoSaurus">
|
||||
LewdManoSaurus</a>.
|
||||
</em>
|
||||
</figurecaption>
|
||||
</figure>
|
||||
<figure>
|
||||
<img alt="Example of a conflict between two mods that both chose the
|
||||
same spot to put a building and tree so they are clipping"
|
||||
src="/img/blog/modmapper-clipping-example3.jpg" />
|
||||
<figurecaption>
|
||||
<em>
|
||||
Conflict between a building and a tree. Screenshot by <a
|
||||
href="https://www.nexusmods.com/skyrimspecialedition/users/51448566">
|
||||
Janquel</a>.
|
||||
</em>
|
||||
</figurecaption>
|
||||
</figure>
|
||||
<figure>
|
||||
<img alt="Example of a conflict between two mods that both chose the
|
||||
same spot to put a woodcutting mill"
|
||||
src="/img/blog/modmapper-clipping-example4.jpg" />
|
||||
<figurecaption>
|
||||
<em>
|
||||
Conflict between two woodcutting mills. Screenshot by <a
|
||||
href="https://www.nexusmods.com/skyrimspecialedition/users/51448566">
|
||||
Janquel</a>.
|
||||
</em>
|
||||
</figurecaption>
|
||||
</figure>
|
||||
</div>
|
||||
</p>
|
||||
|
||||
Mod authors usually use a tool like
|
||||
[TES5Edit](https://www.nexusmods.com/skyrim/mods/25859) to analyze a group of
|
||||
mod plugins to find conflicts and create patches to resolve them on a
|
||||
case-by-case basis. But, I was unsatisfied with that. I wanted to be assured
|
||||
that there would be no conflicts, or at least know the set of all possible mods
|
||||
out there that could conflict so I could manually patch those few mods. There
|
||||
was no good solution for finding conflicts across all mods though. Mod authors
|
||||
would need to download every Skyrim mod ever and no one has time to download all
|
||||
85,000+ Skyrim mods, and no one has the computer memory to load all of those in
|
||||
TES5Edit at the same time.
|
||||
|
||||
Through that frustration, Modmapper was born with the mission to create a
|
||||
database of all Skyrim mod exterior cell edits. With that database I can power
|
||||
the website which visualizes how popular cells are in aggregate as well as allow
|
||||
the user to drill down to individual cells, mods, or plugins to find potential
|
||||
conflicts without ever having to download files themselves.
|
||||
|
||||
When I [released the website about 7 months
|
||||
ago](https://www.reddit.com/r/skyrimmods/comments/sr8k4d/modmapper_over_14_million_cell_edits_from_every/)
|
||||
it made a big splash in the Skyrim modding community. No one had ever visualized
|
||||
mods on a map like this before, and it gave everyone a new perspective on the
|
||||
vast library of Skyrim mods. It was even [featured on the front page of PC
|
||||
Gamer's
|
||||
website](https://www.pcgamer.com/skyrim-modmapper-is-a-weirdly-beautiful-way-to-manage-your-mods/).
|
||||
Thirteen-year-old me, who regularly read the monthly PC Gamer magazine, would
|
||||
have been astounded.
|
||||
|
||||
<a
|
||||
href="https://www.pcgamer.com/skyrim-modmapper-is-a-weirdly-beautiful-way-to-manage-your-mods/"
|
||||
target="_blank">
|
||||
|
||||
</a>
|
||||
|
||||
The comments posted to the initial mod I posted on Nexus Mods[^takedown] for the
|
||||
project were very amusing. It seemed to be blowing their minds:
|
||||
|
||||
> "Quite possibly this could be the best mod for
|
||||
Skyrim. This hands-down makes everyone's life easier to be able to see which of
|
||||
their mods might be conflicting." -- [Nexus Mods comment by
|
||||
lorddonk](/img/blog/modmapper-comment15.png)
|
||||
|
||||
> "The 8th wonder of Skyrim. That's a Titan's work requiring a monk's
|
||||
> perserverance. Finally, a place to go check (in)compatibilities !!! Voted.
|
||||
> Endorsed." -- [Nexus Mods comment by
|
||||
> jfjb2005](/img/blog/modmapper-comment3.png)
|
||||
|
||||
> "They shall sing songs of your greatness! Wow, just wow." -- [Nexus Mods
|
||||
> comment by
|
||||
LumenMystic](/img/blog/modmapper-comment7.png)
|
||||
|
||||
> "Holy Batman Tits! Be honest..... You're a Govt Agent and made this mod during
|
||||
> your "Terrorist Watch Shift" using a CIA super computer.." -- [Nexus Mods
|
||||
comment by toddrizzle](/img/blog/modmapper-comment1.png)
|
||||
|
||||
> "What drugs are you on and can I have some?" -- [Nexus Mods comment by
|
||||
> thappysnek](/img/blog/modmapper-comment11.png)
|
||||
|
||||
> "This is madness! Author are some kind of overhuman?! GREAT work!"-- [Nexus
|
||||
> Mods comment by TeodorWild](/img/blog/modmapper-comment10.png)
|
||||
|
||||
> "You are an absolute legend. Bards will sing tales of your exploits" -- [Nexus
|
||||
> Mods comment by burntwater](/img/blog/modmapper-comment2.png)
|
||||
|
||||
> "I wanted to say something, but I'll just kneel before thee and worship. This
|
||||
> would have taken me a lifetime. Amazing." -- [Nexus Mods comment by
|
||||
> BlueGunk](/img/blog/modmapper-comment8.png)
|
||||
|
||||
> "Finally found the real dragonborn" -- [Nexus Mods comment by
|
||||
> yag1z](/img/blog/modmapper-comment6.png)
|
||||
|
||||
> "he is the messiah!" -- [Nexus Mods comment by
|
||||
> Cursedobjects](/img/blog/modmapper-comment12.png)
|
||||
|
||||
> "A god amongst men." -- [Nexus Mods comment by
|
||||
> TheMotherRobbit](/img/blog/modmapper-comment13.png)
|
||||
|
||||
Apparently knowing how to program is now a god-like ability! This is the type of
|
||||
feedback most programmers aspire to get from their users. I knew the tool was
|
||||
neat and fun to build, but I didn't realize it was *that* sorely needed by the
|
||||
community.
|
||||
|
||||
Today, Modmapper has a sustained user-base of around 7.5k unique visitors a
|
||||
month[^analytics] and I still see it mentioned in reddit threads or discord
|
||||
servers whenever someone is asking about the places a mod edits or what mods
|
||||
might be conflicting in a particular cell.
|
||||
|
||||
The rest of this blog post will delve into how I built the website and how I
|
||||
gathered all of the data necessary to display the visualization.
|
||||
|
||||
### Downloading ALL THE MODS!
|
||||
|
||||
|
||||
|
||||
In order for the project to work I needed to collect all the Skyrim mod plugin
|
||||
files.
|
||||
|
||||
While there are a number of places people upload Skyrim mods, [Nexus
|
||||
Mods](https://nexusmods.com) is conveniently the most popular and has the vast
|
||||
majority of mods. So, I would only need to deal with this one source. Luckily,
|
||||
[they have a nice API
|
||||
handy](https://app.swaggerhub.com/apis-docs/NexusMods/nexus-mods_public_api_params_in_form_data/1.0).
|
||||
|
||||
[modmapper](https://github.com/thallada/modmapper) is the project I created to
|
||||
do this. It is a Rust binary that:
|
||||
|
||||
* Uses [reqwest](https://crates.io/crates/reqwest) to make requests to [Nexus
|
||||
Mods](https://nexusmods.com) for pages of last updated mods.
|
||||
* Uses [scraper](https://crates.io/crates/scraper) to scrape the HTML for
|
||||
individual mod metadata (since the Nexus API doesn't provide an endpoint to
|
||||
list mods).
|
||||
* Makes requests to the Nexus Mods API to get file and download information for
|
||||
each mod, using [serde](https://serde.rs/) to parse the
|
||||
[JSON](https://en.wikipedia.org/wiki/JSON) responses.
|
||||
* Requests the content preview data for each file and walks through the list of
|
||||
files in the archive looking for a Skyrim plugin file (`.esp`, `.esm`, or
|
||||
`.esl`).
|
||||
* If it finds a plugin, it decides to download the mod. It hits the download API
|
||||
to get a download link and downloads the mod file archive.
|
||||
* Then it extracts the archive using one of:
|
||||
[compress_tools](https://crates.io/crates/compress-tools),
|
||||
[unrar](https://crates.io/crates/unrar), or [7zip](https://www.7-zip.org/) via
|
||||
[`std::process::Commmand`](https://doc.rust-lang.org/std/process/struct.Command.html)
|
||||
(depending on what type of archive it is).
|
||||
* With the ESP files (Elder Scrolls Plugin files) extracted, I then use my
|
||||
[skyrim-cell-dump](https://github.com/thallada/skyrim-cell-dump) library (more
|
||||
on that later!) to extract all of the cell edits into structured data.
|
||||
* Uses [seahash](https://crates.io/crates/seahash) to create a fast unique hash
|
||||
for plugin files.
|
||||
* It then saves all of this data to a [postgres](https://www.postgresql.org/)
|
||||
database using the [sqlx crate](https://crates.io/crates/sqlx).
|
||||
* Uses extensive logging with the [tracing
|
||||
crate](https://crates.io/crates/tracing) so I can monitor the output and have
|
||||
a history of a run to debug later if I discover an issue.
|
||||
|
||||
It is designed to be run as a nightly [cron](https://en.wikipedia.org/wiki/Cron)
|
||||
job which downloads mods that have updated on Nexus Mods since the last run.
|
||||
|
||||
To keep costs for this project low, I decided to make the website entirely
|
||||
static. So, instead of creating an API server that would have to be constantly
|
||||
running to serve requests from the website by making queries directly to the
|
||||
database, I would dump all of the data that the website needed from the database
|
||||
to JSON files, then upload those files to [Amazon
|
||||
S3](https://aws.amazon.com/s3/) and serve them through the [Cloudflare
|
||||
CDN](https://www.cloudflare.com/cdn/) which has servers all over the world.
|
||||
|
||||
So, for example, every mod in the database has a JSON file uploaded to
|
||||
`https://mods.modmapper.com/skyrimspecialedition/<nexus_mod_id>.json` and the
|
||||
website frontend will fetch that file when a user clicks a link to that mod in
|
||||
the UI.
|
||||
|
||||
The cost for S3 is pretty reasonable to me ($~3.5/month), and Cloudflare has a
|
||||
[generous free tier](https://www.cloudflare.com/plans/#price-matrix) that allows
|
||||
me to host everything through it for free.
|
||||
|
||||
The server that I actually run `modmapper` on to download all of the mods is a
|
||||
server I already have at home that I also use for other purposes. The output of
|
||||
each run is uploaded to S3, and I also make a full backup of the database and
|
||||
plugin files to [Dropbox](https://www.dropbox.com).
|
||||
|
||||
A lot of people thought it was insane that I downloaded every mod[^adult-mods],
|
||||
but in reality it wasn't too bad once I got all the issues resolved in
|
||||
`modmapper`. I just let it run in the background all day and it would chug
|
||||
through the list of mods one-by-one. Most of the time ended up being spent
|
||||
waiting while the Nexus Mod's API hourly rate limit was reached on my
|
||||
account.[^rate-limit]
|
||||
|
||||
As a result of this project I believe I now have the most complete set of all
|
||||
Skyrim plugins to date (extracted plugins only without other textures, models,
|
||||
etc.)[^plugin-collection]. Compressed, it totals around 99 GB, uncompressed: 191
|
||||
GB.
|
||||
|
||||
[After I downloaded Skyrim Classic mods in addition to Skyrim Special
|
||||
Edition](#finishing-the-collection-by-adding-all-skyrim-classic-mods), here are
|
||||
some counts from the database:
|
||||
|
||||
|:---|:---|
|
||||
| **Mods** | 113,028 |
|
||||
| **Files** | 330,487 |
|
||||
| **Plugins** | 534,831 |
|
||||
| **Plugin Cell Edits** | 33,464,556 |
|
||||
|
||||
### Parsing Skyrim plugin files
|
||||
|
||||
The Skyrim game engine has a concept of
|
||||
[worldspaces](https://en.uesp.net/wiki/Skyrim:Worldspaces) which are exterior
|
||||
areas where the player can travel to. The biggest of these being, of course,
|
||||
Skyrim itself (which, in the lore, is a province of the continent of
|
||||
[Tamriel](https://en.uesp.net/wiki/Lore:Tamriel) on the planet
|
||||
[Nirn](https://en.uesp.net/wiki/Lore:Nirn)). Worldspaces are recorded in a
|
||||
plugin file as [WRLD
|
||||
records](https://en.uesp.net/wiki/Skyrim_Mod:Mod_File_Format/WRLD).
|
||||
|
||||
Worldspaces are then chunked up into a square grid of cells. The Skyrim
|
||||
worldspace consists of a little over 11,000 square cells. Mods that make a
|
||||
changes to the game world have a record in the plugin (a [CELL
|
||||
record](https://en.uesp.net/wiki/Skyrim_Mod:Mod_File_Format/CELL)) with the
|
||||
cell's X and Y coordinates and a list changes in that cell.
|
||||
|
||||
There is some prior art ([esplugin](https://github.com/Ortham/esplugin),
|
||||
[TES5Edit](https://github.com/TES5Edit/TES5Edit),
|
||||
[zedit](https://github.com/z-edit/zedit)) of open-source programs that could
|
||||
parse Skyrim plugins and extract this data. However, all of these were too broad
|
||||
for my purpose or relied on the assumption of being run in the context of a load
|
||||
order where the master files of a plugin would also be available. I wanted a
|
||||
program that could take a single plugin in isolation and skip through all of the
|
||||
non-relevant parts of it and dump just the CELL and WRLD record data plus some
|
||||
metadata about the plugin from the header as fast as possible.
|
||||
|
||||
After discovering [the wonderful documentation on the UESP wiki about the Skyrim
|
||||
mod file format](https://en.uesp.net/wiki/Skyrim_Mod:Mod_File_Format), I
|
||||
realized this would be something that would be possible to make myself.
|
||||
[skyrim-cell-dump](https://github.com/thallada/skyrim-cell-dump) is a Rust
|
||||
library/CLI program that accepts a Skyrim mod file and spits out the header
|
||||
metadata of the plugin, the worlds edited/created, and all of the cells it
|
||||
edits/creates.
|
||||
|
||||
Under the hood, it uses the [nom crate](https://crates.io/crates/nom) to read
|
||||
through the plugin until it finds the relevant records, then uses
|
||||
[flate2](https://crates.io/crates/flate2) to decompress any compressed record
|
||||
data, and finally outputs the extracted data formatted to JSON with
|
||||
[serde](https://crates.io/crates/serde).
|
||||
|
||||
Overall, I was pretty happy with this toolkit of tools and was able to quickly
|
||||
get the data I needed from plugins. My only gripe was that I never quite figured
|
||||
out how to properly do error handling with nom. If there was ever an error, I
|
||||
didn't get much data in the error about what failed besides what function it
|
||||
failed in. I often had to resort to peppering in a dozen `dbg!()` statements to
|
||||
figure out what went wrong.
|
||||
|
||||
I built it as both a library and binary crate so that I could import it in other
|
||||
libraries and get the extracted data directly as Rust structs without needing to
|
||||
go through JSON. I'll go more into why this was useful later.
|
||||
|
||||
### Building the website
|
||||
|
||||
Since I wanted to keep server costs low and wanted the site to be as fast as
|
||||
possible for users, I decided pretty early on that the site would be purely
|
||||
static HTML and JavaScript with no backend server. I decided to use the [Next.js
|
||||
web framework](https://nextjs.org/) with
|
||||
[TypeScript](https://www.typescriptlang.org/) since it was what I was familiar
|
||||
with using in my day job. While it does have [server-side rendering
|
||||
support](https://nextjs.org/docs/basic-features/pages#server-side-rendering)
|
||||
which would require running a backend [Node.js](https://nodejs.org/en/) server,
|
||||
it also supports a limited feature-set that can be [exported as static
|
||||
HTML](https://nextjs.org/docs/advanced-features/static-html-export).
|
||||
|
||||
I host the site on [Cloudflare pages](https://pages.cloudflare.com/) which is
|
||||
available on their free tier and made deploying from Github commits a
|
||||
breeze[^cloudflare]. The web code is in my [modmapper-web
|
||||
repo](https://github.com/thallada/modmapper-web).
|
||||
|
||||
The most prominent feature of the website is the interactive satellite map of
|
||||
Skyrim. Two essential resources made this map possible: [the map tile images
|
||||
from the UESP skyrim map](https://srmap.uesp.net/) and
|
||||
[Mapbox](https://www.mapbox.com/).
|
||||
|
||||
[Mapbox provides a JS library for its WebGL
|
||||
map](https://docs.mapbox.com/mapbox-gl-js/api/) which allows specifying a
|
||||
[raster tile
|
||||
source](https://docs.mapbox.com/mapbox-gl-js/example/map-tiles/)[^3d-terrain].
|
||||
|
||||
The [UESP team painstakingly loaded every cell in the Skyrim worldspace in the
|
||||
Creation Kit and took a
|
||||
screenshot](https://en.uesp.net/wiki/UESPWiki:Skyrim_Map_Design). Once I figured
|
||||
out which image tiles mapped to which in-game cell it was relatively easy to put
|
||||
a map together by plugging them into the Mapbox map as a raster tile source.
|
||||
|
||||
The heatmap overlaid on the map is created using a [Mapbox
|
||||
layer](https://docs.mapbox.com/help/glossary/layer/) that fills a cell with a
|
||||
color on a gradient from green to red depending on how many edits that cell has
|
||||
across the whole database of mods.
|
||||
|
||||
|
||||
|
||||
The sidebar on the site is created using [React](https://reactjs.org/) and
|
||||
[Redux](https://redux.js.org/) and uses the
|
||||
[next/router](https://nextjs.org/docs/api-reference/next/router) to keep track
|
||||
of which page the user is on with URL parameters.
|
||||
|
||||
<p>
|
||||
<div class="row">
|
||||
<img alt="Screenshot of modmapper.com sidebar with a cell selected"
|
||||
src="/img/blog/modmapper-cell-sidebar.jpg" class="half-left" />
|
||||
<img alt="Screenshot of modmapper.com sidebar with a mod selected"
|
||||
src="/img/blog/modmapper-mod-sidebar.jpg" class="half-right" />
|
||||
</div>
|
||||
</p>
|
||||
|
||||
The mod search is implemented using
|
||||
[MiniSearch](https://lucaong.github.io/minisearch/) that asynchronously loads
|
||||
the giant search indices for each game containing every mod name and id.
|
||||
|
||||
|
||||
|
||||
One of the newest features of the site allows users to drill down to a
|
||||
particular plugin within a file of a mod and "Add" it to their list. All of the
|
||||
added plugins will be listed in the sidebar and the cells they edit displayed in
|
||||
purple outlines and conflicts between them displayed in red outlines.
|
||||
|
||||
|
||||
|
||||
### Loading plugins client-side with WebAssembly
|
||||
|
||||
A feature that many users requested after the initial release was being able to
|
||||
load a list of the mods currently installed on their game and see which ones of
|
||||
that set conflict with each other[^second-announcement]. Implementing this
|
||||
feature was one of the most interesting parts of the project. Choosing to use
|
||||
Rust made made it possible, since everything I was running server-side to
|
||||
extract the plugin data could also be done client-side in the browser with the
|
||||
same Rust code compiled to [WebAssembly](https://webassembly.org/).
|
||||
|
||||
I used [wasm-pack](https://github.com/rustwasm/wasm-pack) to create
|
||||
[skyrim-cell-dump-wasm](https://github.com/thallada/skyrim-cell-dump-wasm/)
|
||||
which exported the `parse_plugin` function from my
|
||||
[skyrim-cell-dump](https://github.com/thallada/skyrim-cell-dump) Rust library
|
||||
compiled to WebAssembly. It also exports a `hash_plugin` function that creates a
|
||||
unique hash for a plugin file's slice of bytes using
|
||||
[seahash](https://crates.io/crates/seahash) so the site can link plugins a user
|
||||
has downloaded on their hard-drive to plugins that have been downloaded by
|
||||
modmapper and saved in the database.
|
||||
|
||||
Dragging-and-dropping the Skyrim Data folder on to the webpage or selecting the
|
||||
folder in the "Open Skyrim Data directory" dialog kicks off a process that
|
||||
starts parsing all of the plugin files in that directory in parallel using [Web
|
||||
Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers).
|
||||
|
||||
I developed my own
|
||||
[`WorkerPool`](https://github.com/thallada/modmapper-web/blob/4af628559030c3f24618b29b46d4a40af2f200a6/lib/WorkerPool.ts)
|
||||
that manages creating a pool of available workers and assigns them to plugins to
|
||||
process. The pool size is the number of cores on the user's device so that the
|
||||
site can process as many plugins in parallel as possible. After a plugin
|
||||
finishes processing a plugin and sends the output to the redux store, it gets
|
||||
added back to the pool and is then assigned a new plugin to process if there are
|
||||
any[^wasm-troubles].
|
||||
|
||||
Once all plugins have been loaded, the map updates by displaying all of the
|
||||
cells edited in a purple box and any cells that are edited by more than one
|
||||
plugin in a red box.
|
||||
|
||||
|
||||
|
||||
Users can also drag-and-drop or paste their `plugins.txt` file, which is the
|
||||
file that the game uses to define the load order of plugins and which plugins
|
||||
are enabled or disabled. Adding the `plugins.txt` sorts the list of loaded
|
||||
plugins in the sidebar in load order and enables or disables plugins as defined
|
||||
in the `plugins.txt`.
|
||||
|
||||
|
||||
|
||||
Selecting a cell in the map will display all of the loaded cells that edit that
|
||||
cell in the sidebar.
|
||||
|
||||
|
||||
|
||||
The ability to load plugins straight from a user's hard-drive allows users to
|
||||
map mods that haven't even been uploaded to Nexus Mods.
|
||||
|
||||
### Vortex integration
|
||||
|
||||
The initial mod I released on the Skyrim Special Edition page of Nexus Mods was
|
||||
[taken
|
||||
down](https://www.reddit.com/r/skyrimmods/comments/svnz4a/modmapper_got_removed/)
|
||||
by the site admins since it didn't contain an actual mod and they didn't agree
|
||||
that it qualified as a "Utility".
|
||||
|
||||
Determined to have an actual mod page for Modmapper on Nexus Mods, I decided to
|
||||
make a [Vortex](https://www.nexusmods.com/about/vortex/) integration for
|
||||
modmapper. Vortex is a mod manager made by the developers of Nexus Mods and they
|
||||
allow creating extensions to the tool and have their own [mod section for Vortex
|
||||
extensions](https://www.nexusmods.com/site).
|
||||
|
||||
With the help of [Pickysaurus](https://www.nexusmods.com/skyrim/users/31179975),
|
||||
one of the community managers for Nexus Mods, I created a [Vortex integration
|
||||
for Modmapper](https://www.nexusmods.com/site/mods/371). It adds a context menu
|
||||
option on mods to view the mod in Modmapper with all of the cells it edits
|
||||
selected in purple. It also adds a button next to every plugin file to view just
|
||||
that plugin in Modmapper (assuming it has been processed by Modmapper).
|
||||
|
||||
<p>
|
||||
<div class="row">
|
||||
<img alt="Screenshot of Vortex mod list with a mod context menu open which
|
||||
shows a 'See on Modmapper' option"
|
||||
src="/img/blog/modmapper-vortex-mod-menu.jpg" class="half-left" />
|
||||
<img alt="Screenshot of Vortex plugin list with 'See on Modmapper' buttons
|
||||
on the right of each plugin row"
|
||||
src="/img/blog/modmapper-vortex-plugin-button.jpg" class="half-right" />
|
||||
</div>
|
||||
</p>
|
||||
|
||||
To enable the latter part, I had to include `skyrim-cell-dump-wasm` in the
|
||||
extension so that I could hash the plugin contents with `seahash` to get the
|
||||
same hash that Modmapper would have generated. It only does this hashing when
|
||||
you click the "See on Modmapper" button to save from excessive CPU usage when
|
||||
viewing the plugin list.
|
||||
|
||||
After releasing the Vortex plugin, Pickysaurus [published a news article about
|
||||
modmapper to the Skyrim Special Edition
|
||||
site](https://www.nexusmods.com/skyrimspecialedition/news/14678) which also got
|
||||
a lot of nice comments ❤️.
|
||||
|
||||
### Finishing the collection by adding all Skyrim Classic mods
|
||||
|
||||
Skyrim is very silly in that it has [many
|
||||
editions](https://ag.hyperxgaming.com/article/12043/every-skyrim-edition-released-over-the-last-decade).
|
||||
But there was only one that split the modding universe into two: [Skyrim Special
|
||||
Edition (SE)](https://en.uesp.net/wiki/Skyrim:Special_Edition).
|
||||
It was released in October 2016 with a revamped game engine that brought some
|
||||
sorely needed graphical upgrades. However, it also contained changes to how mods
|
||||
worked, requiring all mod authors to convert their mods to SE. This created big
|
||||
chasm in the library of mods, and Nexus Mods had to make a separate section for
|
||||
SE-only mods.
|
||||
|
||||
When I started downloading mods in 2021, I started only with Skyrim SE mods,
|
||||
which, at the time of writing, totals at over [55,000 mods on Nexus
|
||||
Mods](https://www.nexusmods.com/skyrimspecialedition/mods/).
|
||||
|
||||
After releasing with just SE mods, many users requested that all of the classic
|
||||
pre-SE Skyrim mods be added as well. This month, I finally finished downloading
|
||||
all Skyrim Classic mods, which, at the time of writing, totals at over [68,000
|
||||
mods on Nexus Mods](https://www.nexusmods.com/skyrim/mods/). That brings the
|
||||
total downloaded and processed mods for Modmapper at over 113,000
|
||||
mods[^adult-mods]!
|
||||
|
||||
### The future
|
||||
|
||||
A lot of users had great feedback and suggestions on what to add to the site. I
|
||||
could only implement so many of them, though. The rest I've been keeping track
|
||||
of on [this Trello board](https://trello.com/b/VdpTQ7ar/modmapper).
|
||||
|
||||
Some of the headline items on it are:
|
||||
|
||||
* Add [Solstheim map](https://dbmap.uesp.net/)
|
||||
|
||||
Since map tiles images are available for that worldspace and because I have
|
||||
already recorded edits to the worldspace in my database, it shouldn't be too
|
||||
terribly difficult.
|
||||
* Add [Mod Organizer 2](https://www.modorganizer.org/) plugin
|
||||
|
||||
Lots of people requested this since it's a very popular mod manager compared
|
||||
to Vortex. MO2 supports python extensions so I created
|
||||
[skyrim-cell-dump-py](https://github.com/thallada/skyrim-cell-dump-py) to
|
||||
export the Rust plugin processing code to a Python library. I got a bit stuck
|
||||
on actually creating the plugin though, so it might be a while until I get to
|
||||
that.
|
||||
* Find a way to display interior cell edits on the map
|
||||
|
||||
The map is currently missing edits to interior cells. Since almost all
|
||||
interior cells in Skyrim have a link to the exterior world through a door
|
||||
teleporter, it should be possible to map an interior cell edit to an exterior
|
||||
cell on the map based on which cell the door leads out to.
|
||||
|
||||
That will require digging much more into the plugin files for more data, and
|
||||
city worldspaces will complicate things further. Then there's the question of
|
||||
interiors with multiple doors to different exterior cells, or interior cells
|
||||
nested recursively deep within many other interior cells.
|
||||
* Create a standalone [Electron](https://www.electronjs.org) app that can run
|
||||
outside the browser
|
||||
|
||||
I think this would solve a lot of the issues I ran into while developing the
|
||||
website. Since Electron has a Node.js process running on the user's computer
|
||||
outside the sandboxed browser process, it gives me much more flexibility. It
|
||||
could do things like automatically load a user's plugin files. Or just load
|
||||
plugins at all wihtout having to deal with the annoying dialog that lies to
|
||||
the user saying they're about to upload their entire Data folder hundreds of
|
||||
gigabytes full of files to a server (I really wish the
|
||||
[HTMLInputElement.webkitdirectory](https://developer.mozilla.org/en-US/docs/Web/API/HTMLInputElement/webkitdirectory)
|
||||
API would use the same underlying code as the [HTML Drag and Drop
|
||||
API](https://developer.mozilla.org/en-US/docs/Web/API/HTML_Drag_and_Drop_API)
|
||||
which is a lot better).
|
||||
* Improving the search
|
||||
|
||||
The mod search feature struggles the most with the static generated nature of
|
||||
the site. I found it very hard to pack all of the necessary info for the
|
||||
search index for all 100k+ mods (index for both SE and LE is around 6 MB).
|
||||
Asynchronously loading the indices with MiniSearch keeps it from freezing up
|
||||
the browser, but it does take a very long time to fully load. I can't help
|
||||
think that there's a better way to shard the indices somehow and only fetch
|
||||
what I need based on what the user is typing into the search.
|
||||
|
||||
To be clear, a lot of the Todos on the board are pipe-dreams. I may never get to
|
||||
them. This project is sustained purely by my motivation and self-interests and
|
||||
if something is too much of a pain to get working I'll just drop it.
|
||||
|
||||
There will also be future Elder Scrolls games, and [future Bethesda games based
|
||||
on roughly the same game engine](https://bethesda.net/en/game/starfield). It
|
||||
would be neat to create similar database for those games as the modding
|
||||
community develops in realtime.
|
||||
|
||||
Overall, I'm glad I made something of use to the modding community. I hope to
|
||||
keep the site running for as long as people are modding Skyrim (until the
|
||||
heat-death of the universe, probably).
|
||||
|
||||
|
||||
<br>
|
||||
|
||||
---
|
||||
|
||||
#### Footnotes
|
||||
|
||||
[^bazaarrealm]:
|
||||
Unfortunately, I basically lost interest on the mod after working on
|
||||
Modmapper. I might still write a blog post about it eventually since I did a
|
||||
lot of interesting hacking on the Skyrim game engine to try to add some
|
||||
asynchronous multiplayer aspects. [Project is here if anyone is curious in
|
||||
the meantime](https://github.com/thallada/BazaarRealmPlugin).
|
||||
|
||||
[^takedown]:
|
||||
I sadly only have screenshots for some of the comments on that mod since it
|
||||
was eventually taken down by the Nexus Mod admins. See explanation about
|
||||
that in the [Vortex integration section](#vortex-integration).
|
||||
|
||||
[^analytics]:
|
||||
As recorded by Cloudflare's server side analytics, which may record a fair
|
||||
amount of bot traffic. I suspect this is the most accurate number I can get
|
||||
since most of my users probably use an ad blocker that blocks client-side
|
||||
analytics.
|
||||
|
||||
[^adult-mods]:
|
||||
Every mod on Nexus Mods except for adult mods since the site restricts
|
||||
viewing adult mods to only logged-in users and I wasn't able to get my
|
||||
scraping bot to log in as a user.
|
||||
|
||||
[^rate-limit]:
|
||||
Apparently my mass-downloading did not go unnoticed by the Nexus Mod admins.
|
||||
I think it's technically against their terms of service to automatically
|
||||
download mods, but I somehow got on their good side and was spared the
|
||||
ban-hammer. I don't recommend anyone else run modmapper themselves on the
|
||||
entire site unless you talk to the admins beforehand and get the okay from
|
||||
them.
|
||||
|
||||
[^plugin-collection]:
|
||||
If you would like access to this dataset of plugins to do some data-mining
|
||||
please reach out to me at [tyler@hallada.net](mailto:tyler@hallada.net)
|
||||
(Note: only contains plugins files, no models, textures, audio, etc.). I
|
||||
don't plan on releasing it publicly since that would surely go against many
|
||||
mod authors' wishes/licenses.
|
||||
|
||||
[^cloudflare]:
|
||||
I'm not sure I want to recommend anyone else use Cloudflare after [the whole
|
||||
Kiwi Farms
|
||||
debacle](https://www.theverge.com/2022/9/6/23339889/cloudflare-kiwi-farms-content-moderation-ddos).
|
||||
I now regret having invested so much of the infrastructure in them. However,
|
||||
I'm only using their free-tier, so at least I am a net-negative for their
|
||||
business? I would recommend others look into
|
||||
[Netlify](https://www.netlify.com/) or [fastly](https://www.fastly.com/) for
|
||||
similar offerings to Cloudflare pages/CDN.
|
||||
|
||||
[^3d-terrain]:
|
||||
I also tried to add a [raster Terrain-DEM
|
||||
source](https://docs.mapbox.com/data/tilesets/reference/mapbox-terrain-dem-v1/)
|
||||
for rendering the terrain in 3D. I got fairly far [generating my own DEM RGB
|
||||
tiles](https://github.com/syncpoint/terrain-rgb) from an upscaled [greyscale
|
||||
heightmap](https://i.imgur.com/9RErBDo.png) [constructed from the LAND
|
||||
records in Skyrim.esm](https://www.nexusmods.com/skyrim/mods/80692) (view it
|
||||
[here](https://www.dropbox.com/s/56lffk021riil6h/heightmap-4x_foolhardy_Remacri_rgb.tif?dl=0)).
|
||||
But, it came out all wrong: [giant cliffs in the middle of the
|
||||
map](/img/blog/modmapper-terrain-cliff.jpg) and [tiny spiky lumps with big
|
||||
jumps in elevation at cell boundaries](/img/blog/modmapper-bad-terrain.jpg).
|
||||
Seemed like too much work to get right than it was worth it.
|
||||
|
||||
[^second-announcement]:
|
||||
[This was the announcement I posted to /r/skyrimmods for this feature](
|
||||
https://www.reddit.com/r/skyrimmods/comments/ti3gjh/modmapper_update_load_plugins_in_your_load_order/)
|
||||
|
||||
[^wasm-troubles]:
|
||||
At first, I noticed a strange issue with re-using the same worker on
|
||||
different plugins multiple times. After a while (~30 reuses per worker), the
|
||||
processing would slow to a crawl and eventually strange things started
|
||||
happening (I was listening to music in my browser and it started to pop and
|
||||
crack). It seemed like the speed of processing increased exponentially to
|
||||
the number of times the worker was reused. So, to avoid this, I had to make
|
||||
the worker pool terminate and recreate workers after every plugin processed.
|
||||
This ended up not being as slow as it sounds and worked fine. However, I
|
||||
recently discovered that [wee_alloc, the most suggested allocator to use
|
||||
with rust in wasm, has a memory leak and is mostly unmaintained
|
||||
now](https://www.reddit.com/r/rust/comments/x1cle0/dont_use_wee_alloc_in_production_code_targeting/).
|
||||
I switched to the default allocator and I didn't run into the exponentially
|
||||
slow re-use problem. For some reason, the first run on a fresh tab is always
|
||||
much faster than the second run, but subsequent runs are still fairly stable
|
||||
in processing time.
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 52 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 52 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 815 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 160 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 857 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.8 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 373 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 504 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 335 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 336 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 656 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 151 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.0 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 692 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 247 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 329 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 42 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 307 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 101 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 92 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 62 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 283 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 423 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 324 KiB |
BIN
assets/me.png
BIN
assets/me.png
Binary file not shown.
|
Before Width: | Height: | Size: 412 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 349 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 828 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 848 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 99 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 311 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 372 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 3.0 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 410 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1017 KiB |
BIN
assets/nyc.jpg
BIN
assets/nyc.jpg
Binary file not shown.
|
Before Width: | Height: | Size: 745 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 763 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 472 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 336 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 227 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 630 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 538 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.4 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 471 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 332 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 269 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 720 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 463 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 337 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 130 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.5 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.5 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 621 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 551 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.7 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 428 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 388 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.2 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 368 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 293 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 448 KiB |
@ -4,42 +4,40 @@ title: Tyler Hallada - Blog
|
||||
---
|
||||
|
||||
{% for post in paginator.posts %}
|
||||
{% if post.hidden == null or post.hidden == false %}
|
||||
<div class="card">
|
||||
<div class="row clearfix">
|
||||
<div class="column full post-header">
|
||||
<h2 class="post-title"><a href="{{ post.url }}">{{ post.title }}</a></h2>
|
||||
<span class="timestamp">{{ post.date | date_to_string }}</span>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="row clearfix post-header">
|
||||
<div class="column three-fourths">
|
||||
<a href="{{ post.url }}"><h2 class="post-title">{{ post.title }}</h2></a>
|
||||
</div>
|
||||
<div class="row clearfix">
|
||||
<div class="column full post">
|
||||
{{ post.excerpt }}
|
||||
</div>
|
||||
</div>
|
||||
<div class="row clearfix more-row">
|
||||
<div class="column full">
|
||||
<a href="{{ post.url }}" class="read-more">Read More »</a>
|
||||
</div>
|
||||
<div class="column fourth">
|
||||
<span class="timestamp">{{ post.date | date_to_string }}</span>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
<div class="row clearfix">
|
||||
<div class="column full post">
|
||||
{{ post.excerpt }}
|
||||
</div>
|
||||
</div>
|
||||
<div class="row clearfix more-row">
|
||||
<div class="column full">
|
||||
<a href="{{ post.url }}" class="read-more">Read More »</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{% endfor %}
|
||||
|
||||
<div class="row clearfix pagination">
|
||||
<div class="column full">
|
||||
{% if paginator.previous_page %}
|
||||
{% if paginator.page == 2 %}
|
||||
<a href="/blog/" class="previous">« Previous</a>
|
||||
<a href="/blog" class="previous">« Previous</a>
|
||||
{% else %}
|
||||
<a href="/blog/page{{ paginator.previous_page }}/" class="previous">« Previous</a>
|
||||
<a href="/blog/page{{ paginator.previous_page }}" class="previous">« Previous</a>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
<span class="page_number ">Page {{ paginator.page }} of {{ paginator.total_pages }}</span>
|
||||
{% if paginator.next_page %}
|
||||
<a href="/blog/page{{ paginator.next_page }}/" class="next">Next »</a>
|
||||
<a href="/blog/page{{ paginator.next_page }}" class="next">Next »</a>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% include mail-form.html %}
|
||||
|
||||
184
css/main.css
184
css/main.css
@ -13,21 +13,12 @@
|
||||
}
|
||||
|
||||
html {
|
||||
background-color: whitesmoke;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: whitesmoke;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
min-height: 100vh;
|
||||
background-color: whitesmoke;
|
||||
}
|
||||
|
||||
.root {
|
||||
font-family: 'Open Sans', Arial, sans-serif;
|
||||
font-style: normal;
|
||||
font-weight: 400;
|
||||
@ -48,39 +39,9 @@ a {
|
||||
}
|
||||
|
||||
p {
|
||||
margin: 0 0 12px;
|
||||
font-size: 1rem;
|
||||
line-height: 1.45rem;
|
||||
margin: 0 0 10px;
|
||||
}
|
||||
|
||||
img + em {
|
||||
display: block;
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
li {
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
|
||||
table {
|
||||
min-width: 85%;
|
||||
margin: 1em auto 1em;
|
||||
border: 1px solid #cbcbcb;
|
||||
}
|
||||
|
||||
td, th {
|
||||
padding: 0.5em 0.5em;
|
||||
}
|
||||
|
||||
td {
|
||||
border: 1px solid #cbcbcb;
|
||||
}
|
||||
|
||||
tr:nth-child(2n-1) td {
|
||||
background-color: #f2f2f2;
|
||||
}
|
||||
|
||||
|
||||
a:visited {
|
||||
color:#855C85;
|
||||
}
|
||||
@ -128,14 +89,10 @@ blockquote p {
|
||||
.container {
|
||||
margin: 0 auto;
|
||||
max-width: 48rem;
|
||||
width: 100%;
|
||||
width: 90%;
|
||||
}
|
||||
|
||||
@media (min-width: 40rem) {
|
||||
.container {
|
||||
width: 90%;
|
||||
}
|
||||
|
||||
.column {
|
||||
float: left;
|
||||
padding-left: 1rem;
|
||||
@ -148,8 +105,6 @@ blockquote p {
|
||||
.column.third { width: 33.3%; }
|
||||
.column.fourth { width: 25%; }
|
||||
.column.three-fourths { width: 75%; }
|
||||
.column.fifth { width: 20%; }
|
||||
.column.four-fifths { width: 80%; }
|
||||
.column.flow-opposite { float: right; }
|
||||
}
|
||||
|
||||
@ -203,15 +158,17 @@ img { width: auto; max-width: 100%; height: auto; }
|
||||
.hide-mobile {
|
||||
display: none;
|
||||
}
|
||||
.hide-desktop-inline-block { display: inline-block }
|
||||
.hide-desktop-block { display: block }
|
||||
.hide-desktop-inline { display: inline }
|
||||
.hide-desktop-inline-block {
|
||||
display: inline-block;
|
||||
}
|
||||
.hide-desktop-block {
|
||||
display: block;
|
||||
}
|
||||
|
||||
@media (min-width: 40rem) {
|
||||
.hide-desktop { display: none }
|
||||
.hide-mobile-inline-block { display: inline-block }
|
||||
.hide-mobile-block { display: block }
|
||||
.hide-mobile-inline { display: inline }
|
||||
}
|
||||
|
||||
/*****************************************************************************/
|
||||
@ -226,12 +183,10 @@ h1.title {
|
||||
font-size: 2rem;
|
||||
font-weight: 200;
|
||||
margin: 0;
|
||||
margin-left: 0.5rem;
|
||||
}
|
||||
|
||||
div.header {
|
||||
/* this is padding instead of margin to prevent <html> from poking out behind top of page */
|
||||
padding: 20px 0 20px;
|
||||
margin: 20px 0 20px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
@ -246,18 +201,18 @@ div.header a {
|
||||
}
|
||||
|
||||
span.timestamp {
|
||||
display: block;
|
||||
font-size: 0.85rem;
|
||||
margin-top: 8px;
|
||||
margin-top: 7px;
|
||||
}
|
||||
|
||||
@media (min-width: 40rem) {
|
||||
span.timestamp {
|
||||
float: right;
|
||||
}
|
||||
}
|
||||
|
||||
.post-header {
|
||||
padding-bottom: 24px;
|
||||
display: flex;
|
||||
align-items: start;
|
||||
justify-content: space-between;
|
||||
column-gap: 12px;
|
||||
flex-wrap: wrap;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
a.read-more {
|
||||
@ -287,7 +242,7 @@ div.more-row {
|
||||
|
||||
div.pagination {
|
||||
text-align: center;
|
||||
margin-bottom: 20px;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
div.rss {
|
||||
@ -362,11 +317,6 @@ a.rss img {
|
||||
background-color: #333;
|
||||
}
|
||||
|
||||
.post img {
|
||||
display: block;
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
.post img.half-left {
|
||||
float: none;
|
||||
width: 100%;
|
||||
@ -390,58 +340,6 @@ a.rss img {
|
||||
}
|
||||
}
|
||||
|
||||
.post .row {
|
||||
display: flex;
|
||||
align: center;
|
||||
justify-content: center;
|
||||
gap: 8px 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.post .row figure {
|
||||
flex-basis: calc(50% - 8px);
|
||||
}
|
||||
|
||||
@media (max-width: 40rem) {
|
||||
.post .row {
|
||||
flex-direction: column;
|
||||
}
|
||||
}
|
||||
|
||||
.post figure {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.post figure figurecaption {
|
||||
display: block;
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
/*****************************************************************************/
|
||||
/*
|
||||
/* Subscribe form
|
||||
/*
|
||||
/*****************************************************************************/
|
||||
|
||||
.subscribe-form h3 {
|
||||
margin-top: 10px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
|
||||
.subscribe-form input {
|
||||
margin: 10px;
|
||||
}
|
||||
|
||||
.subscribe-form label {
|
||||
margin-left: 10px;
|
||||
}
|
||||
|
||||
.subscribe-form span.form-rss {
|
||||
display: block;
|
||||
margin-top: 20px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
|
||||
/*****************************************************************************/
|
||||
/*
|
||||
/* Homepage
|
||||
@ -464,6 +362,7 @@ a.rss img {
|
||||
background: white;
|
||||
margin: 0 0.5rem 1rem;
|
||||
border-radius: 3px;
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
h1.big-name {
|
||||
@ -594,42 +493,3 @@ div.options-panel form {
|
||||
div.options-panel form label {
|
||||
display: block;
|
||||
}
|
||||
|
||||
/*****************************************************************************/
|
||||
/*
|
||||
/* Comments (isso)
|
||||
/*
|
||||
/*****************************************************************************/
|
||||
|
||||
.isso-postbox .textarea-wrapper {
|
||||
margin-top: 10px;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
div.isso-postbox div.form-wrapper section.auth-section p.input-wrapper {
|
||||
margin-right: 10px;
|
||||
}
|
||||
|
||||
/*****************************************************************************/
|
||||
/*
|
||||
/* Light & Dark Theme Toggle
|
||||
/*
|
||||
/*****************************************************************************/
|
||||
|
||||
div.theme-toggle {
|
||||
position: static;
|
||||
bottom: 0;
|
||||
width: 100%;
|
||||
text-align: center;
|
||||
z-index: 2;
|
||||
padding: 0.25rem 0.75rem;
|
||||
}
|
||||
|
||||
@media (min-width: 40rem) {
|
||||
div.theme-toggle {
|
||||
position: absolute;
|
||||
top: 0;
|
||||
right: 0;
|
||||
width: inherit;
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,40 +0,0 @@
|
||||
/* Dark theme */
|
||||
|
||||
html {
|
||||
filter: invert(90%);
|
||||
background-color: rgba(10, 10, 10, 0.9);
|
||||
}
|
||||
|
||||
img:not(.icon), video, div.video-container {
|
||||
filter: invert(100%);
|
||||
}
|
||||
|
||||
img:not(.icon) {
|
||||
opacity: .90;
|
||||
transition: opacity .5s ease-in-out;
|
||||
}
|
||||
|
||||
img:hover:not(.icon) {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
a:not(.card), a:hover:not(.card) {
|
||||
filter: invert(100%);
|
||||
}
|
||||
|
||||
a:not(.card) img:not(.icon) {
|
||||
filter: none;
|
||||
}
|
||||
|
||||
.card {
|
||||
box-shadow: 0 1px 2px #fff !important;
|
||||
}
|
||||
|
||||
.post pre {
|
||||
background-color: #fff !important;
|
||||
}
|
||||
|
||||
.post a code {
|
||||
background-color: #222 !important;
|
||||
border: 1px solid #333 !important;
|
||||
}
|
||||
@ -1,56 +0,0 @@
|
||||
.highlight code {
|
||||
filter: invert(100%);
|
||||
background-color: #000 !important;
|
||||
}
|
||||
.highlight { background: #0F0908; color: #E0CCAE; }
|
||||
.highlight .c { color: #6B4035; } /* Comment */
|
||||
.highlight .err { color: #F2DDBC; background-color: #66292F } /* Error */
|
||||
.highlight .o { color: #A67458 } /* Operator */
|
||||
.highlight .cm { color: #6B4035; } /* Comment.Multiline */
|
||||
.highlight .cp { color: #6B4035; } /* Comment.Preproc */
|
||||
.highlight .c1 { color: #6B4035; } /* Comment.Single */
|
||||
.highlight .cs { color: #6B4035; } /* Comment.Special */
|
||||
.highlight .gd { color: #BF472C; background-color: #291916 } /* Generic.Deleted */
|
||||
.highlight .gd .x { color: #000000; background-color: #ffaaaa } /* Generic.Deleted.Specific */
|
||||
.highlight .gr { color: #66292F } /* Generic.Error */
|
||||
.highlight .gh { color: #F2A766 } /* Generic.Heading */
|
||||
.highlight .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */
|
||||
.highlight .gi .x { color: #000000; background-color: #aaffaa } /* Generic.Inserted.Specific */
|
||||
.highlight .go { color: #E0CCAE } /* Generic.Output */
|
||||
.highlight .gp { color: #8A4B53 } /* Generic.Prompt */
|
||||
.highlight .gu { color: #F2A766 } /* Generic.Subheading */
|
||||
.highlight .gt { color: #BF472C } /* Generic.Traceback */
|
||||
.highlight .kt { color: #BF472C } /* Keyword.Type */
|
||||
.highlight .m { color: #8A4B53 } /* Literal.Number */
|
||||
.highlight .s { color: #D47D49 } /* Literal.String */
|
||||
.highlight .na { color: #BF472C } /* Name.Attribute */
|
||||
.highlight .nb { color: #F2A766 } /* Name.Builtin */
|
||||
.highlight .nc { color: #445588 } /* Name.Class */
|
||||
.highlight .no { color: #F2DDBC } /* Name.Constant */
|
||||
.highlight .ni { color: #a4896f } /* Name.Entity */
|
||||
.highlight .ne { color: #990000 } /* Name.Exception */
|
||||
.highlight .nf { color: #BF472C } /* Name.Function */
|
||||
.highlight .nn { color: #BF472C } /* Name.Namespace */
|
||||
.highlight .nt { color: #F2A766 } /* Name.Tag */
|
||||
.highlight .nv { color: #A67458 } /* Name.Variable */
|
||||
.highlight .w { color: #bbbbbb } /* Text.Whitespace */
|
||||
.highlight .mf { color: #8A4B53 } /* Literal.Number.Float */
|
||||
.highlight .mh { color: #8A4B53 } /* Literal.Number.Hex */
|
||||
.highlight .mi { color: #8A4B53 } /* Literal.Number.Integer */
|
||||
.highlight .mo { color: #8A4B53 } /* Literal.Number.Oct */
|
||||
.highlight .sb { color: #D47D49 } /* Literal.String.Backtick */
|
||||
.highlight .sc { color: #D47D49 } /* Literal.String.Char */
|
||||
.highlight .sd { color: #D47D49 } /* Literal.String.Doc */
|
||||
.highlight .s2 { color: #D47D49 } /* Literal.String.Double */
|
||||
.highlight .se { color: #D47D49 } /* Literal.String.Escape */
|
||||
.highlight .sh { color: #D47D49 } /* Literal.String.Heredoc */
|
||||
.highlight .si { color: #D47D49 } /* Literal.String.Interpol */
|
||||
.highlight .sx { color: #D47D49 } /* Literal.String.Other */
|
||||
.highlight .sr { color: #D47D49 } /* Literal.String.Regex */
|
||||
.highlight .s1 { color: #D47D49 } /* Literal.String.Single */
|
||||
.highlight .ss { color: #D47D49 } /* Literal.String.Symbol */
|
||||
.highlight .bp { color: #999999 } /* Name.Builtin.Pseudo */
|
||||
.highlight .vc { color: #A67458 } /* Name.Variable.Class */
|
||||
.highlight .vg { color: #A67458 } /* Name.Variable.Global */
|
||||
.highlight .vi { color: #A67458 } /* Name.Variable.Instance */
|
||||
.highlight .il { color: #8A4B53 } /* Literal.Number.Integer.Long */
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user