diff --git a/_drafts/neural-style.md b/_drafts/neural-style.md new file mode 100644 index 0000000..e862371 --- /dev/null +++ b/_drafts/neural-style.md @@ -0,0 +1,275 @@ +--- +title: Generating Realistic Satellite Imagery with Deep Neural Networks +layout: post +--- + +I've been doing a lot of experimenting with [neural-style](https://github.com/jcjohnson/neural-style) +the last month. I think I've discovered a few exciting applications of the +technique that I haven't seen anyone else do yet. The true power of this +algorithm really shines when you can see concrete examples. + +Skip to the **Applications** part of this post to see the outputs from my +experimentation if you are already familiar with DeepDream, Deep Style, and all +the other latest research in generating images with deep neural networks. + +###Background and History### + +On [May 18, 2015 at 2 a.m., Alexander +Mordvintsev](https://medium.com/backchannel/inside-deep-dreams-how-google-made-its-computers-go-crazy-83b9d24e66df#.g4t69y8wy), +an engineer at Google, did something with deep neural networks that no one had +done before. He took a net designed for *recognizing* objects in images and used +it to *generate* objects in images. In a sense, he was telling these systems +that mimic the human visual cortex to hallucinate things that weren't really +there. The [results](https://i.imgur.com/6ocuQsZ.jpg) looked remarkably like LSD +trips or what a [schizophrenic person sees on a blank +wall](https://www.reddit.com/r/deepdream/comments/3cewgn/an_artist_suffering_from_schizophrenia_was_told/). + +Mordvintsev's discovery quickly gathered attention at Google once he posted +images from his experimentation on the company's internal network. On June 17, +2015, [Google posted a blog post about the +technique](http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html) +(dubbed "Inceptionism") and how it was useful for opening up the notoriously +black-boxed neural networks using visualizations that researchers could examine. +These machine hallucinations were key for identifying the features of objects +that neural networks used to tell one object from another (like a dog from a +cat). But the post also revealed the [beautiful +results](https://goo.gl/photos/fFcivHZ2CDhqCkZdA) of applying the algorithm +iteratively on it's own outputs and zooming out at each step. + +The internet exploded in response to this post. And once [Google posted the code +for performing the +technique](http://googleresearch.blogspot.com/2015/07/deepdream-code-example-for-visualizing.html?m=1), +people began experimenting and sharing [their fantastic and creepy +images](https://www.reddit.com/r/deepdream) with the world. + +Then, on August, 26, 2015, a paper titled ["A Neural Algorithm of Artistic +Style"](http://arxiv.org/abs/1508.06576) was published. It showed how one could +identify which layers of deep neural networks recognized stylistic information +of an image (and not the content) and then use this stylistic information in +Google's Inceptionism technique to paint other images in the style of any +artist. A [few](https://github.com/jcjohnson/neural-style) +[implementations](https://github.com/kaishengtai/neuralart) of the paper were +put up on Github. This exploded the internet again in a frenzy. This time, the +images produced were less like psychedelic-induced nightmares but more like the +next generation of Instagram filters ([reddit +how-to](https://www.reddit.com/r/deepdream/comments/3jwl76/how_anyone_can_create_deep_style_images/)). + +People began to wonder [what all of this +meant](http://www.hopesandfears.com/hopes/culture/is-this-art/215039-deep-dream-google-art) +to [the future of +art](http://kajsotala.fi/2015/07/deepdream-today-psychedelic-images-tomorrow-unemployed-artists/). +Some of the results produced where [indistinguishable from the style of dead +artists' +works](https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/outputs/tubingen_starry.png). +Was this a demonstration of creativity in computers or just a neat trick? + +On November, 19, 2015, [another paper](http://arxiv.org/abs/1511.06434) was +released that demonstrated a technique for generating scenes from convolutional +neural nets ([implementation on Github](https://github.com/Newmu/dcgan_code)). +The program could generate random (and very realistic) [bedroom +images](https://github.com/Newmu/dcgan_code/raw/master/images/lsun_bedrooms_five_epoch_samples.png) +from a neural net trained on bedroom images. Amazingly, it could also generate +[the same bedroom from any +angle](https://github.com/Newmu/dcgan_code/blob/master/images/lsun_bedrooms_five_epochs_interps.png). +It could also [produce images of the same procedurally generated face from any +angle](https://github.com/Newmu/dcgan_code/blob/master/images/turn_vector.png). +Theoretically, we could use this technology to create *procedurally generated +game art*. + +The main thing holding this technology back from revolutionizing procedurally +generated video games is that it is not real-time. Using +[neural-style](https://github.com/jcjohnson/neural-style) to apply artistic +style to a 512 by 512 pixel content image could take minutes even on the +top-of-the-line GTX Titan X graphics card. Still, I believe this technology has +a lot of potential for generating game art even if it can't act as a real-time +filter. + +###Applications: Generating Satellite Images for Procedural World Maps### + +I personally know very little machine learning, but I have been able to produce +a lot of interesting results by using the tool provided by +[neural-style](https://github.com/jcjohnson/neural-style). + +Inspired by [Kaelan's procedurally generated world +maps](http://blog.kaelan.org/randomly-generated-world-map/), I wanted to extend +the idea by generating realistic satellite images of the terrain maps. The +procedure is simple: take a [generated terrain map](/assets/kaelan_terrain1.png) +and apply the style of a [real-world satellite image](/assets/uk_satellite.jpg) +on it using neural-style. + +![Output of generated map plus real-world satellite +imagery](/assets/satellite_terrain1_process.png) + +The generated output takes on whatever terrain is in the satellite image. Here +is an output processing one of Kaelan's maps with a [arctic satellite +image](/assets/svalbard_satellite.jpg): + +![Kaelan's terrain map](/assets/kaelan_terrain2.jpg) +![Output of terrain map plus arctic satellite imagery](/assets/satellite_terrain2.png) + +And again, with one of Kaelan's desert maps and a [satellite image of a +desert](/assets/desert_satellite.jpg): + +![Kaelan's desert terrain map](/assets/kaelan_terrain3.jpg) +![Output of terrain map plus desert satellite imagery](/assets/satellite_terrain3.png) + +It even works with [Kaelan's generated hexagon +maps](http://blog.kaelan.org/hexagon-world-map-generation/). Here's an island +hexagon map plus a [satellite image of a volcanic +island](/assets/volcano_satellite.jpg): + +![Kaelan's island hexagon map](/assets/kaelan_hex_terrain.jpg) +![Output of hexagon map plus island satellite +imagery](/assets/satellite_hex_terrain.png) + +This image even produced an interesting three-dimensional effect because of the +volcano in the satellite image. + +By the way, this also works with minecraft maps. Here's a minecraft map I found +on the internet plus a [satellite image from Google +Earth](/assets/river_satellite.png): + +![Minecraft map](/assets/minecraft_map.jpg) +![Output of minecraft map plus river satellite +imagery](/assets/satellite_minecraft_map.png) + +No fancy texture packs or 3-D rendering needed :). + +Here is the Fallout 4 grayscale map plus a +[satellite image of Boston](/assets/boston_aerial.jpg): + +![Fallout 4 grayscale map](/assets/fallout4_map.png) +![Output of Fallout 4 map plus Boston satellite +imagery](/assets/satellite_fallout4_map.png) + +Unfortunately, it puts the built-up dense part of the city in the wrong part of +the geographic area. But, this is understandable since we gave the algorithm no +information on where that is on the map. + +We can also make the generated terrain maps look like old hand-drawn maps using +neural-style. With Kaelan's terrain map as the +content and [the in-game Elder Scrolls IV Oblivion map of +Cyrodiil](/assets/cyrodiil_ingame.jpg) as the style we get this: + +![Kaelan's terrain map](/assets/kaelan_terrain1.png) +![Output of terrain map plus map of Cyrodiil](/assets/cyrodiil_terrain1.png) + +It looks cool, but the water isn't conveyed very clearly (e.g. makes deep water +look like land). Neural-style seems to work better when there is lots of color +in both images. + +Here is the output of the hex terrain plus satellite map above and the Cyrodiil +map which looks a little cleaner: + +![Satellite-like hex terrain map](/assets/satellite_hex_terrain.png) +![Output of hex terrain plus satellite and map of +Cyrodiil](/assets/cyrodiil_satellite_hex_terrain.png) + +I was interested to see what neural-style could generate from random noise, so I +rendered some clouds in GIMP and ran it with a satellite image of [Mexico City +from Google Earth](/assets/mexico_city.jpg) (by the way, I've been getting high +quality Google Earth shots from +[earthview.withgoogle.com](https://earthview.withgoogle.com)). + +![Random clouds](/assets/blurry_clouds.png) +![Output of random clouds and Mexico City](/assets/random_mexico_city.png) + +Not bad for a neural net without a degree in urban planning. + +I also tried generating on random noise with a satellite image of [a water +treatment plant in Peru](/assets/treatment_plant.jpg) + +![Random clouds](/assets/blurry_clouds2.png) +![Output of random clouds and water treatment +plant](/assets/random_treatment_plant.png) + +###Applications: More Fun### + +For fun, here are some other outputs that I liked. + +[My photo of Boston's skyline as the content](/assets/boston_skyline.jpg) and +[Vincent van Gogh's The Starry Night as the style](/assets/starry_night.jpg): + +![Output of Boston skyline and starry night](/assets/starry_boston.png) + +[A photo of me](/assets/standing_forest.jpg) (by Aidan Bevacqua) and [Forrest in +the end of Autumn by Caspar David Friedrich](/assets/forrest_autumn.jpg): + +![Output of me and Forrest in the end of +Autumn](/assets/dead_forest_standing.png) + +[Another photo of me by Aidan](/assets/sitting_forest.jpg) in the same style: + +![Output of me and Forrest in the end of Autumn](/assets/dead_forest_sitting.png) + +[A photo of me on a mountain](/assets/mountain_view.jpg) (by Aidan Bevacqua) and +[pixel art by Paul Robertson](/assets/pixels.png) + +![Output of me on a mountain and pixel art](/assets/mountain_view_pixels.png) + +[A photo of a park in Copenhagen I took](/assets/copenhagen_park.jpg) and a +painting similar in composition, [Avenue of Poplars at Sunset by Vincent van +Gogh](/assets/avenue_poplars.jpg): + +![Output of park in Copenhagen and Avenue of Poplars at +Sunset](/assets/poplars.png) + +[My photo of the Shenandoah National Park](/assets/shenandoah_mountains.jpg) and +[this halo graphic from GMUNK](/assets/halo_ring_mountains.jpg) +([GMUNK](http://www.gmunk.com/filter/Interactive/ORA-Summoners-HALO)): + +![Output of Shenandoah mountains and halo ring +mountains](/assets/halo_shenandoah.png) + +[A photo of me by Aidan](/assets/me.png) and a [stained glass +fractal](/assets/stained_glass.jpg): + +![Output of me and a stained glass fractal](/assets/stained_glass_portrait.png) + +Same photo of me and some [psychedelic art by GMUNK](/assets/pockets.jpg) + +![Output of me and psychedelic art](/assets/pockets_portrait.png) + +[New York City](/assets/nyc.jpg) and [a rainforest](/assets/rainforest.jpg): + +![Output of New York City and a rainforest](/assets/jungle_nyc.png) + +[Kowloon Walled City](/assets/kowloon.jpg) and [a National Geographic +Map](/assets/ngs_map.jpg): + +![Output of Kowloon and NGS map](/assets/kowloon_ngs.png) + +[A photo of me by Aidan](/assets/side_portrait.jpg) and [Head of Lioness by +Theodore Gericault](/assets/head_lioness.jpg): + +![Output of photo of me and ](/assets/lion_portrait.png) + +[Photo I took of a Norwegian forest](/assets/forest_hill.jpg) and [The Mountain +Brook by Albert Bierstadt](/assets/mountain_brook.jpg): + +![Output of Norwegian forest and The Mountain +Brook](/assets/mountain_brook_hill.png) + +###Limitations### + +I don't have infinite money for a GTX Titan X, so I'm stuck with using OpenCL on +my more-than-a-few-generations-old AMD card. It takes about a half-hour to +generate one 512x512 px image in my set-up (which makes the feedback loop for +correcting mistakes *very* long). And sometimes the neural-style refuses to run +on my GPU (I suspect it runs out of VRAM), so I have to run it on my CPU which +takes even longer... + +I am unable to generate bigger images (though +[the author has been able to generate up to 1920x1010 +px](https://github.com/jcjohnson/neural-style/issues/36#issuecomment-142994812)). +As the size of the output increases the amount of memory and time to generate +also increases. And, it's not practical to just generate thumbnails to test +parameters, because increasing the image size will probably generate a very +different image since all the other parameters stay the same even though they +are dependent on the image size. + +Some people have had success running these neural nets on GPU spot instances in +AWS. It would be certainly cheaper than buying a new GPU in the short-term. + +So, I have a few more ideas for what to run, but it will take me quite a while +to get through the queue. diff --git a/assets/avenue_poplars.jpg b/assets/avenue_poplars.jpg new file mode 100644 index 0000000..cfc1527 Binary files /dev/null and b/assets/avenue_poplars.jpg differ diff --git a/assets/blurry_clouds.png b/assets/blurry_clouds.png new file mode 100644 index 0000000..0b519c5 Binary files /dev/null and b/assets/blurry_clouds.png differ diff --git a/assets/blurry_clouds2.png b/assets/blurry_clouds2.png new file mode 100644 index 0000000..d07154a Binary files /dev/null and b/assets/blurry_clouds2.png differ diff --git a/assets/boston_aerial.jpg b/assets/boston_aerial.jpg new file mode 100644 index 0000000..95bf774 Binary files /dev/null and b/assets/boston_aerial.jpg differ diff --git a/assets/boston_skyline.jpg b/assets/boston_skyline.jpg new file mode 100644 index 0000000..8ee9878 Binary files /dev/null and b/assets/boston_skyline.jpg differ diff --git a/assets/copenhagen_park.jpg b/assets/copenhagen_park.jpg new file mode 100644 index 0000000..b44f2fc Binary files /dev/null and b/assets/copenhagen_park.jpg differ diff --git a/assets/cyrodiil_ingame.jpg b/assets/cyrodiil_ingame.jpg new file mode 100644 index 0000000..0a99694 Binary files /dev/null and b/assets/cyrodiil_ingame.jpg differ diff --git a/assets/cyrodiil_satellite_hex_terrain.png b/assets/cyrodiil_satellite_hex_terrain.png new file mode 100644 index 0000000..e0b338a Binary files /dev/null and b/assets/cyrodiil_satellite_hex_terrain.png differ diff --git a/assets/cyrodiil_terrain1.png b/assets/cyrodiil_terrain1.png new file mode 100644 index 0000000..10b9c63 Binary files /dev/null and b/assets/cyrodiil_terrain1.png differ diff --git a/assets/dead_forest_sitting.png b/assets/dead_forest_sitting.png new file mode 100644 index 0000000..cbb39d5 Binary files /dev/null and b/assets/dead_forest_sitting.png differ diff --git a/assets/dead_forest_standing.png b/assets/dead_forest_standing.png new file mode 100644 index 0000000..0a187b0 Binary files /dev/null and b/assets/dead_forest_standing.png differ diff --git a/assets/desert_satellite.jpg b/assets/desert_satellite.jpg new file mode 100755 index 0000000..e4250c3 Binary files /dev/null and b/assets/desert_satellite.jpg differ diff --git a/assets/fallout4_map.png b/assets/fallout4_map.png new file mode 100644 index 0000000..b66a8e3 Binary files /dev/null and b/assets/fallout4_map.png differ diff --git a/assets/forest_hill.jpg b/assets/forest_hill.jpg new file mode 100644 index 0000000..5dcbfb1 Binary files /dev/null and b/assets/forest_hill.jpg differ diff --git a/assets/forrest_autumn.jpg b/assets/forrest_autumn.jpg new file mode 100644 index 0000000..d77858b Binary files /dev/null and b/assets/forrest_autumn.jpg differ diff --git a/assets/halo_ring_mountains.jpg b/assets/halo_ring_mountains.jpg new file mode 100644 index 0000000..d7dfe2f Binary files /dev/null and b/assets/halo_ring_mountains.jpg differ diff --git a/assets/halo_shenandoah.png b/assets/halo_shenandoah.png new file mode 100644 index 0000000..b814f2c Binary files /dev/null and b/assets/halo_shenandoah.png differ diff --git a/assets/head_lioness.jpg b/assets/head_lioness.jpg new file mode 100644 index 0000000..e080d1c Binary files /dev/null and b/assets/head_lioness.jpg differ diff --git a/assets/jungle_nyc.png b/assets/jungle_nyc.png new file mode 100644 index 0000000..74e47fe Binary files /dev/null and b/assets/jungle_nyc.png differ diff --git a/assets/kaelan_hex_terrain.jpg b/assets/kaelan_hex_terrain.jpg new file mode 100644 index 0000000..1e19868 Binary files /dev/null and b/assets/kaelan_hex_terrain.jpg differ diff --git a/assets/kaelan_terrain1.png b/assets/kaelan_terrain1.png new file mode 100644 index 0000000..b6995f4 Binary files /dev/null and b/assets/kaelan_terrain1.png differ diff --git a/assets/kaelan_terrain2.jpg b/assets/kaelan_terrain2.jpg new file mode 100644 index 0000000..2234e40 Binary files /dev/null and b/assets/kaelan_terrain2.jpg differ diff --git a/assets/kaelan_terrain3.jpg b/assets/kaelan_terrain3.jpg new file mode 100644 index 0000000..5ef8bca Binary files /dev/null and b/assets/kaelan_terrain3.jpg differ diff --git a/assets/kowloon.jpg b/assets/kowloon.jpg new file mode 100644 index 0000000..9f924f7 Binary files /dev/null and b/assets/kowloon.jpg differ diff --git a/assets/kowloon_ngs.png b/assets/kowloon_ngs.png new file mode 100644 index 0000000..7dcb93d Binary files /dev/null and b/assets/kowloon_ngs.png differ diff --git a/assets/lion_portrait.png b/assets/lion_portrait.png new file mode 100644 index 0000000..2ee458f Binary files /dev/null and b/assets/lion_portrait.png differ diff --git a/assets/me.png b/assets/me.png new file mode 100644 index 0000000..32f0b4d Binary files /dev/null and b/assets/me.png differ diff --git a/assets/mexico_city.jpg b/assets/mexico_city.jpg new file mode 100644 index 0000000..3c5a01f Binary files /dev/null and b/assets/mexico_city.jpg differ diff --git a/assets/minecraft_map.jpg b/assets/minecraft_map.jpg new file mode 100644 index 0000000..d49ba1c Binary files /dev/null and b/assets/minecraft_map.jpg differ diff --git a/assets/mountain_brook.jpg b/assets/mountain_brook.jpg new file mode 100644 index 0000000..837edab Binary files /dev/null and b/assets/mountain_brook.jpg differ diff --git a/assets/mountain_brook_hill.png b/assets/mountain_brook_hill.png new file mode 100644 index 0000000..02d2230 Binary files /dev/null and b/assets/mountain_brook_hill.png differ diff --git a/assets/mountain_view.jpg b/assets/mountain_view.jpg new file mode 100644 index 0000000..87d678c Binary files /dev/null and b/assets/mountain_view.jpg differ diff --git a/assets/mountain_view_pixels.png b/assets/mountain_view_pixels.png new file mode 100644 index 0000000..e7d68e5 Binary files /dev/null and b/assets/mountain_view_pixels.png differ diff --git a/assets/ngs_map.jpg b/assets/ngs_map.jpg new file mode 100644 index 0000000..eb8a0ad Binary files /dev/null and b/assets/ngs_map.jpg differ diff --git a/assets/nyc.jpg b/assets/nyc.jpg new file mode 100644 index 0000000..3b9091a Binary files /dev/null and b/assets/nyc.jpg differ diff --git a/assets/pixels.png b/assets/pixels.png new file mode 100644 index 0000000..a18f696 Binary files /dev/null and b/assets/pixels.png differ diff --git a/assets/pockets.jpg b/assets/pockets.jpg new file mode 100644 index 0000000..792c59c Binary files /dev/null and b/assets/pockets.jpg differ diff --git a/assets/pockets_portrait.png b/assets/pockets_portrait.png new file mode 100644 index 0000000..64c2940 Binary files /dev/null and b/assets/pockets_portrait.png differ diff --git a/assets/poplars.png b/assets/poplars.png new file mode 100644 index 0000000..1eb5450 Binary files /dev/null and b/assets/poplars.png differ diff --git a/assets/rainforest.jpg b/assets/rainforest.jpg new file mode 100644 index 0000000..e9ed2e5 Binary files /dev/null and b/assets/rainforest.jpg differ diff --git a/assets/random_mexico_city.png b/assets/random_mexico_city.png new file mode 100644 index 0000000..e166f2c Binary files /dev/null and b/assets/random_mexico_city.png differ diff --git a/assets/random_treatment_plant.png b/assets/random_treatment_plant.png new file mode 100644 index 0000000..5d799ad Binary files /dev/null and b/assets/random_treatment_plant.png differ diff --git a/assets/river_satellite.png b/assets/river_satellite.png new file mode 100644 index 0000000..7bc855f Binary files /dev/null and b/assets/river_satellite.png differ diff --git a/assets/satellite_fallout4_map.png b/assets/satellite_fallout4_map.png new file mode 100644 index 0000000..f107e43 Binary files /dev/null and b/assets/satellite_fallout4_map.png differ diff --git a/assets/satellite_hex_terrain.png b/assets/satellite_hex_terrain.png new file mode 100644 index 0000000..fb4801b Binary files /dev/null and b/assets/satellite_hex_terrain.png differ diff --git a/assets/satellite_minecraft_map.png b/assets/satellite_minecraft_map.png new file mode 100644 index 0000000..25c6d95 Binary files /dev/null and b/assets/satellite_minecraft_map.png differ diff --git a/assets/satellite_terrain1_process.png b/assets/satellite_terrain1_process.png new file mode 100644 index 0000000..b4d8450 Binary files /dev/null and b/assets/satellite_terrain1_process.png differ diff --git a/assets/satellite_terrain2.png b/assets/satellite_terrain2.png new file mode 100644 index 0000000..3fb09fd Binary files /dev/null and b/assets/satellite_terrain2.png differ diff --git a/assets/satellite_terrain3.png b/assets/satellite_terrain3.png new file mode 100755 index 0000000..ae6e1aa Binary files /dev/null and b/assets/satellite_terrain3.png differ diff --git a/assets/shenandoah_mountains.jpg b/assets/shenandoah_mountains.jpg new file mode 100644 index 0000000..ed9730e Binary files /dev/null and b/assets/shenandoah_mountains.jpg differ diff --git a/assets/side_portrait.jpg b/assets/side_portrait.jpg new file mode 100644 index 0000000..a916e23 Binary files /dev/null and b/assets/side_portrait.jpg differ diff --git a/assets/sitting_forest.jpg b/assets/sitting_forest.jpg new file mode 100644 index 0000000..d4f11cf Binary files /dev/null and b/assets/sitting_forest.jpg differ diff --git a/assets/stained_glass.jpg b/assets/stained_glass.jpg new file mode 100644 index 0000000..21868fe Binary files /dev/null and b/assets/stained_glass.jpg differ diff --git a/assets/stained_glass_portrait.png b/assets/stained_glass_portrait.png new file mode 100644 index 0000000..f6e7d13 Binary files /dev/null and b/assets/stained_glass_portrait.png differ diff --git a/assets/standing_forest.jpg b/assets/standing_forest.jpg new file mode 100644 index 0000000..2fa6928 Binary files /dev/null and b/assets/standing_forest.jpg differ diff --git a/assets/starry_boston.png b/assets/starry_boston.png new file mode 100644 index 0000000..45a5f51 Binary files /dev/null and b/assets/starry_boston.png differ diff --git a/assets/starry_night.jpg b/assets/starry_night.jpg new file mode 100644 index 0000000..724457a Binary files /dev/null and b/assets/starry_night.jpg differ diff --git a/assets/svalbard_satellite.jpg b/assets/svalbard_satellite.jpg new file mode 100644 index 0000000..f2a595e Binary files /dev/null and b/assets/svalbard_satellite.jpg differ diff --git a/assets/treatment_plant.jpg b/assets/treatment_plant.jpg new file mode 100644 index 0000000..91c440e Binary files /dev/null and b/assets/treatment_plant.jpg differ diff --git a/assets/uk_satellite.jpg b/assets/uk_satellite.jpg new file mode 100644 index 0000000..4a08c31 Binary files /dev/null and b/assets/uk_satellite.jpg differ diff --git a/assets/volcano_satellite.jpg b/assets/volcano_satellite.jpg new file mode 100644 index 0000000..7575927 Binary files /dev/null and b/assets/volcano_satellite.jpg differ