Browse Source

New blog draft

Tyler Hallada 8 years ago
parent
commit
dd7418be14
62 changed files with 275 additions and 0 deletions
  1. 275 0
      _drafts/neural-style.md
  2. BIN
      assets/avenue_poplars.jpg
  3. BIN
      assets/blurry_clouds.png
  4. BIN
      assets/blurry_clouds2.png
  5. BIN
      assets/boston_aerial.jpg
  6. BIN
      assets/boston_skyline.jpg
  7. BIN
      assets/copenhagen_park.jpg
  8. BIN
      assets/cyrodiil_ingame.jpg
  9. BIN
      assets/cyrodiil_satellite_hex_terrain.png
  10. BIN
      assets/cyrodiil_terrain1.png
  11. BIN
      assets/dead_forest_sitting.png
  12. BIN
      assets/dead_forest_standing.png
  13. BIN
      assets/desert_satellite.jpg
  14. BIN
      assets/fallout4_map.png
  15. BIN
      assets/forest_hill.jpg
  16. BIN
      assets/forrest_autumn.jpg
  17. BIN
      assets/halo_ring_mountains.jpg
  18. BIN
      assets/halo_shenandoah.png
  19. BIN
      assets/head_lioness.jpg
  20. BIN
      assets/jungle_nyc.png
  21. BIN
      assets/kaelan_hex_terrain.jpg
  22. BIN
      assets/kaelan_terrain1.png
  23. BIN
      assets/kaelan_terrain2.jpg
  24. BIN
      assets/kaelan_terrain3.jpg
  25. BIN
      assets/kowloon.jpg
  26. BIN
      assets/kowloon_ngs.png
  27. BIN
      assets/lion_portrait.png
  28. BIN
      assets/me.png
  29. BIN
      assets/mexico_city.jpg
  30. BIN
      assets/minecraft_map.jpg
  31. BIN
      assets/mountain_brook.jpg
  32. BIN
      assets/mountain_brook_hill.png
  33. BIN
      assets/mountain_view.jpg
  34. BIN
      assets/mountain_view_pixels.png
  35. BIN
      assets/ngs_map.jpg
  36. BIN
      assets/nyc.jpg
  37. BIN
      assets/pixels.png
  38. BIN
      assets/pockets.jpg
  39. BIN
      assets/pockets_portrait.png
  40. BIN
      assets/poplars.png
  41. BIN
      assets/rainforest.jpg
  42. BIN
      assets/random_mexico_city.png
  43. BIN
      assets/random_treatment_plant.png
  44. BIN
      assets/river_satellite.png
  45. BIN
      assets/satellite_fallout4_map.png
  46. BIN
      assets/satellite_hex_terrain.png
  47. BIN
      assets/satellite_minecraft_map.png
  48. BIN
      assets/satellite_terrain1_process.png
  49. BIN
      assets/satellite_terrain2.png
  50. BIN
      assets/satellite_terrain3.png
  51. BIN
      assets/shenandoah_mountains.jpg
  52. BIN
      assets/side_portrait.jpg
  53. BIN
      assets/sitting_forest.jpg
  54. BIN
      assets/stained_glass.jpg
  55. BIN
      assets/stained_glass_portrait.png
  56. BIN
      assets/standing_forest.jpg
  57. BIN
      assets/starry_boston.png
  58. BIN
      assets/starry_night.jpg
  59. BIN
      assets/svalbard_satellite.jpg
  60. BIN
      assets/treatment_plant.jpg
  61. BIN
      assets/uk_satellite.jpg
  62. BIN
      assets/volcano_satellite.jpg

+ 275 - 0
_drafts/neural-style.md

@@ -0,0 +1,275 @@
1
+---
2
+title: Generating Realistic Satellite Imagery with Deep Neural Networks
3
+layout: post
4
+---
5
+
6
+I've been doing a lot of experimenting with [neural-style](https://github.com/jcjohnson/neural-style)
7
+the last month. I think I've discovered a few exciting applications of the
8
+technique that I haven't seen anyone else do yet. The true power of this
9
+algorithm really shines when you can see concrete examples.
10
+
11
+Skip to the **Applications** part of this post to see the outputs from my
12
+experimentation if you are already familiar with DeepDream, Deep Style, and all
13
+the other latest research in generating images with deep neural networks.
14
+
15
+###Background and History###
16
+
17
+On [May 18, 2015 at 2 a.m., Alexander
18
+Mordvintsev](https://medium.com/backchannel/inside-deep-dreams-how-google-made-its-computers-go-crazy-83b9d24e66df#.g4t69y8wy),
19
+an engineer at Google, did something with deep neural networks that no one had
20
+done before. He took a net designed for *recognizing* objects in images and used
21
+it to *generate* objects in images. In a sense, he was telling these systems
22
+that mimic the human visual cortex to hallucinate things that weren't really
23
+there. The [results](https://i.imgur.com/6ocuQsZ.jpg) looked remarkably like LSD
24
+trips or what a [schizophrenic person sees on a blank
25
+wall](https://www.reddit.com/r/deepdream/comments/3cewgn/an_artist_suffering_from_schizophrenia_was_told/).
26
+
27
+Mordvintsev's discovery quickly gathered attention at Google once he posted
28
+images from his experimentation on the company's internal network. On June 17,
29
+2015, [Google posted a blog post about the
30
+technique](http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html)
31
+(dubbed "Inceptionism") and how it was useful for opening up the notoriously
32
+black-boxed neural networks using visualizations that researchers could examine.
33
+These machine hallucinations were key for identifying the features of objects
34
+that neural networks used to tell one object from another (like a dog from a
35
+cat). But the post also revealed the [beautiful
36
+results](https://goo.gl/photos/fFcivHZ2CDhqCkZdA) of applying the algorithm
37
+iteratively on it's own outputs and zooming out at each step.
38
+
39
+The internet exploded in response to this post. And once [Google posted the code
40
+for performing the
41
+technique](http://googleresearch.blogspot.com/2015/07/deepdream-code-example-for-visualizing.html?m=1),
42
+people began experimenting and sharing [their fantastic and creepy
43
+images](https://www.reddit.com/r/deepdream) with the world.
44
+
45
+Then, on August, 26, 2015, a paper titled ["A Neural Algorithm of Artistic
46
+Style"](http://arxiv.org/abs/1508.06576) was published. It showed how one could
47
+identify which layers of deep neural networks recognized stylistic information
48
+of an image (and not the content) and then use this stylistic information in
49
+Google's Inceptionism technique to paint other images in the style of any
50
+artist. A [few](https://github.com/jcjohnson/neural-style)
51
+[implementations](https://github.com/kaishengtai/neuralart) of the paper were
52
+put up on Github. This exploded the internet again in a frenzy. This time, the
53
+images produced were less like psychedelic-induced nightmares but more like the
54
+next generation of Instagram filters ([reddit
55
+how-to](https://www.reddit.com/r/deepdream/comments/3jwl76/how_anyone_can_create_deep_style_images/)). 
56
+
57
+People began to wonder [what all of this
58
+meant](http://www.hopesandfears.com/hopes/culture/is-this-art/215039-deep-dream-google-art)
59
+to [the future of
60
+art](http://kajsotala.fi/2015/07/deepdream-today-psychedelic-images-tomorrow-unemployed-artists/).
61
+Some of the results produced where [indistinguishable from the style of dead
62
+artists'
63
+works](https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/outputs/tubingen_starry.png).
64
+Was this a demonstration of creativity in computers or just a neat trick?
65
+
66
+On November, 19, 2015, [another paper](http://arxiv.org/abs/1511.06434) was
67
+released that demonstrated a technique for generating scenes from convolutional
68
+neural nets ([implementation on Github](https://github.com/Newmu/dcgan_code)).
69
+The program could generate random (and very realistic) [bedroom
70
+images](https://github.com/Newmu/dcgan_code/raw/master/images/lsun_bedrooms_five_epoch_samples.png)
71
+from a neural net trained on bedroom images. Amazingly, it could also generate
72
+[the same bedroom from any
73
+angle](https://github.com/Newmu/dcgan_code/blob/master/images/lsun_bedrooms_five_epochs_interps.png).
74
+It could also [produce images of the same procedurally generated face from any
75
+angle](https://github.com/Newmu/dcgan_code/blob/master/images/turn_vector.png).
76
+Theoretically, we could use this technology to create *procedurally generated
77
+game art*.
78
+
79
+The main thing holding this technology back from revolutionizing procedurally
80
+generated video games is that it is not real-time. Using
81
+[neural-style](https://github.com/jcjohnson/neural-style) to apply artistic
82
+style to a 512 by 512 pixel content image could take minutes even on the
83
+top-of-the-line GTX Titan X graphics card. Still, I believe this technology has
84
+a lot of potential for generating game art even if it can't act as a real-time
85
+filter.
86
+
87
+###Applications: Generating Satellite Images for Procedural World Maps###
88
+
89
+I personally know very little machine learning, but I have been able to produce
90
+a lot of interesting results by using the tool provided by
91
+[neural-style](https://github.com/jcjohnson/neural-style).
92
+
93
+Inspired by [Kaelan's procedurally generated world
94
+maps](http://blog.kaelan.org/randomly-generated-world-map/), I wanted to extend
95
+the idea by generating realistic satellite images of the terrain maps. The
96
+procedure is simple: take a [generated terrain map](/assets/kaelan_terrain1.png)
97
+and apply the style of a [real-world satellite image](/assets/uk_satellite.jpg)
98
+on it using neural-style.
99
+
100
+![Output of generated map plus real-world satellite
101
+imagery](/assets/satellite_terrain1_process.png)
102
+
103
+The generated output takes on whatever terrain is in the satellite image. Here
104
+is an output processing one of Kaelan's maps with a [arctic satellite
105
+image](/assets/svalbard_satellite.jpg):
106
+
107
+![Kaelan's terrain map](/assets/kaelan_terrain2.jpg)
108
+![Output of terrain map plus arctic satellite imagery](/assets/satellite_terrain2.png)
109
+
110
+And again, with one of Kaelan's desert maps and a [satellite image of a
111
+desert](/assets/desert_satellite.jpg):
112
+
113
+![Kaelan's desert terrain map](/assets/kaelan_terrain3.jpg)
114
+![Output of terrain map plus desert satellite imagery](/assets/satellite_terrain3.png)
115
+
116
+It even works with [Kaelan's generated hexagon
117
+maps](http://blog.kaelan.org/hexagon-world-map-generation/). Here's an island
118
+hexagon map plus a [satellite image of a volcanic
119
+island](/assets/volcano_satellite.jpg):
120
+
121
+![Kaelan's island hexagon map](/assets/kaelan_hex_terrain.jpg)
122
+![Output of hexagon map plus island satellite
123
+imagery](/assets/satellite_hex_terrain.png)
124
+
125
+This image even produced an interesting three-dimensional effect because of the
126
+volcano in the satellite image.
127
+
128
+By the way, this also works with minecraft maps. Here's a minecraft map I found
129
+on the internet plus a [satellite image from Google
130
+Earth](/assets/river_satellite.png):
131
+
132
+![Minecraft map](/assets/minecraft_map.jpg)
133
+![Output of minecraft map plus river satellite
134
+imagery](/assets/satellite_minecraft_map.png)
135
+
136
+No fancy texture packs or 3-D rendering needed :).
137
+
138
+Here is the Fallout 4 grayscale map plus a
139
+[satellite image of Boston](/assets/boston_aerial.jpg):
140
+
141
+![Fallout 4 grayscale map](/assets/fallout4_map.png)
142
+![Output of Fallout 4 map plus Boston satellite
143
+imagery](/assets/satellite_fallout4_map.png)
144
+
145
+Unfortunately, it puts the built-up dense part of the city in the wrong part of
146
+the geographic area. But, this is understandable since we gave the algorithm no
147
+information on where that is on the map.
148
+
149
+We can also make the generated terrain maps look like old hand-drawn maps using
150
+neural-style. With Kaelan's terrain map as the
151
+content and [the in-game Elder Scrolls IV Oblivion map of
152
+Cyrodiil](/assets/cyrodiil_ingame.jpg) as the style we get this:
153
+
154
+![Kaelan's terrain map](/assets/kaelan_terrain1.png)
155
+![Output of terrain map plus map of Cyrodiil](/assets/cyrodiil_terrain1.png)
156
+
157
+It looks cool, but the water isn't conveyed very clearly (e.g. makes deep water
158
+look like land). Neural-style seems to work better when there is lots of color
159
+in both images.
160
+
161
+Here is the output of the hex terrain plus satellite map above and the Cyrodiil
162
+map which looks a little cleaner:
163
+
164
+![Satellite-like hex terrain map](/assets/satellite_hex_terrain.png)
165
+![Output of hex terrain plus satellite and map of
166
+Cyrodiil](/assets/cyrodiil_satellite_hex_terrain.png)
167
+
168
+I was interested to see what neural-style could generate from random noise, so I
169
+rendered some clouds in GIMP and ran it with a satellite image of [Mexico City
170
+from Google Earth](/assets/mexico_city.jpg) (by the way, I've been getting high
171
+quality Google Earth shots from
172
+[earthview.withgoogle.com](https://earthview.withgoogle.com)).
173
+
174
+![Random clouds](/assets/blurry_clouds.png)
175
+![Output of random clouds and Mexico City](/assets/random_mexico_city.png)
176
+
177
+Not bad for a neural net without a degree in urban planning.
178
+
179
+I also tried generating on random noise with a satellite image of [a water
180
+treatment plant in Peru](/assets/treatment_plant.jpg)
181
+
182
+![Random clouds](/assets/blurry_clouds2.png)
183
+![Output of random clouds and water treatment
184
+plant](/assets/random_treatment_plant.png)
185
+
186
+###Applications: More Fun###
187
+
188
+For fun, here are some other outputs that I liked.
189
+
190
+[My photo of Boston's skyline as the content](/assets/boston_skyline.jpg) and
191
+[Vincent van Gogh's The Starry Night as the style](/assets/starry_night.jpg):
192
+
193
+![Output of Boston skyline and starry night](/assets/starry_boston.png)
194
+
195
+[A photo of me](/assets/standing_forest.jpg) (by Aidan Bevacqua) and [Forrest in
196
+the end of Autumn by Caspar David Friedrich](/assets/forrest_autumn.jpg):
197
+
198
+![Output of me and Forrest in the end of
199
+Autumn](/assets/dead_forest_standing.png)
200
+
201
+[Another photo of me by Aidan](/assets/sitting_forest.jpg) in the same style:
202
+
203
+![Output of me and Forrest in the end of Autumn](/assets/dead_forest_sitting.png)
204
+
205
+[A photo of me on a mountain](/assets/mountain_view.jpg) (by Aidan Bevacqua) and
206
+[pixel art by Paul Robertson](/assets/pixels.png)
207
+
208
+![Output of me on a mountain and pixel art](/assets/mountain_view_pixels.png)
209
+
210
+[A photo of a park in Copenhagen I took](/assets/copenhagen_park.jpg) and a
211
+painting similar in composition, [Avenue of Poplars at Sunset by Vincent van
212
+Gogh](/assets/avenue_poplars.jpg):
213
+
214
+![Output of park in Copenhagen and Avenue of Poplars at
215
+Sunset](/assets/poplars.png)
216
+
217
+[My photo of the Shenandoah National Park](/assets/shenandoah_mountains.jpg) and
218
+[this halo graphic from GMUNK](/assets/halo_ring_mountains.jpg)
219
+([GMUNK](http://www.gmunk.com/filter/Interactive/ORA-Summoners-HALO)):
220
+
221
+![Output of Shenandoah mountains and halo ring
222
+mountains](/assets/halo_shenandoah.png)
223
+
224
+[A photo of me by Aidan](/assets/me.png) and a [stained glass
225
+fractal](/assets/stained_glass.jpg):
226
+
227
+![Output of me and a stained glass fractal](/assets/stained_glass_portrait.png)
228
+
229
+Same photo of me and some [psychedelic art by GMUNK](/assets/pockets.jpg)
230
+
231
+![Output of me and psychedelic art](/assets/pockets_portrait.png)
232
+
233
+[New York City](/assets/nyc.jpg) and [a rainforest](/assets/rainforest.jpg):
234
+
235
+![Output of New York City and a rainforest](/assets/jungle_nyc.png)
236
+
237
+[Kowloon Walled City](/assets/kowloon.jpg) and [a National Geographic
238
+Map](/assets/ngs_map.jpg):
239
+
240
+![Output of Kowloon and NGS map](/assets/kowloon_ngs.png)
241
+
242
+[A photo of me by Aidan](/assets/side_portrait.jpg) and [Head of Lioness by
243
+Theodore Gericault](/assets/head_lioness.jpg):
244
+
245
+![Output of photo of me and ](/assets/lion_portrait.png)
246
+
247
+[Photo I took of a Norwegian forest](/assets/forest_hill.jpg) and [The Mountain
248
+Brook by Albert Bierstadt](/assets/mountain_brook.jpg):
249
+
250
+![Output of Norwegian forest and The Mountain
251
+Brook](/assets/mountain_brook_hill.png)
252
+
253
+###Limitations###
254
+
255
+I don't have infinite money for a GTX Titan X, so I'm stuck with using OpenCL on
256
+my more-than-a-few-generations-old AMD card. It takes about a half-hour to
257
+generate one 512x512 px image in my set-up (which makes the feedback loop for
258
+correcting mistakes *very* long). And sometimes the neural-style refuses to run
259
+on my GPU (I suspect it runs out of VRAM), so I have to run it on my CPU which
260
+takes even longer...
261
+
262
+I am unable to generate bigger images (though
263
+[the author has been able to generate up to 1920x1010
264
+px](https://github.com/jcjohnson/neural-style/issues/36#issuecomment-142994812)).
265
+As the size of the output increases the amount of memory and time to generate
266
+also increases. And, it's not practical to just generate thumbnails to test
267
+parameters, because increasing the image size will probably generate a very
268
+different image since all the other parameters stay the same even though they
269
+are dependent on the image size.
270
+
271
+Some people have had success running these neural nets on GPU spot instances in
272
+AWS. It would be certainly cheaper than buying a new GPU in the short-term.
273
+
274
+So, I have a few more ideas for what to run, but it will take me quite a while
275
+to get through the queue.

BIN
assets/avenue_poplars.jpg


BIN
assets/blurry_clouds.png


BIN
assets/blurry_clouds2.png


BIN
assets/boston_aerial.jpg


BIN
assets/boston_skyline.jpg


BIN
assets/copenhagen_park.jpg


BIN
assets/cyrodiil_ingame.jpg


BIN
assets/cyrodiil_satellite_hex_terrain.png


BIN
assets/cyrodiil_terrain1.png


BIN
assets/dead_forest_sitting.png


BIN
assets/dead_forest_standing.png


BIN
assets/desert_satellite.jpg


BIN
assets/fallout4_map.png


BIN
assets/forest_hill.jpg


BIN
assets/forrest_autumn.jpg


BIN
assets/halo_ring_mountains.jpg


BIN
assets/halo_shenandoah.png


BIN
assets/head_lioness.jpg


BIN
assets/jungle_nyc.png


BIN
assets/kaelan_hex_terrain.jpg


BIN
assets/kaelan_terrain1.png


BIN
assets/kaelan_terrain2.jpg


BIN
assets/kaelan_terrain3.jpg


BIN
assets/kowloon.jpg


BIN
assets/kowloon_ngs.png


BIN
assets/lion_portrait.png


BIN
assets/me.png


BIN
assets/mexico_city.jpg


BIN
assets/minecraft_map.jpg


BIN
assets/mountain_brook.jpg


BIN
assets/mountain_brook_hill.png


BIN
assets/mountain_view.jpg


BIN
assets/mountain_view_pixels.png


BIN
assets/ngs_map.jpg


BIN
assets/nyc.jpg


BIN
assets/pixels.png


BIN
assets/pockets.jpg


BIN
assets/pockets_portrait.png


BIN
assets/poplars.png


BIN
assets/rainforest.jpg


BIN
assets/random_mexico_city.png


BIN
assets/random_treatment_plant.png


BIN
assets/river_satellite.png


BIN
assets/satellite_fallout4_map.png


BIN
assets/satellite_hex_terrain.png


BIN
assets/satellite_minecraft_map.png


BIN
assets/satellite_terrain1_process.png


BIN
assets/satellite_terrain2.png


BIN
assets/satellite_terrain3.png


BIN
assets/shenandoah_mountains.jpg


BIN
assets/side_portrait.jpg


BIN
assets/sitting_forest.jpg


BIN
assets/stained_glass.jpg


BIN
assets/stained_glass_portrait.png


BIN
assets/standing_forest.jpg


BIN
assets/starry_boston.png


BIN
assets/starry_night.jpg


BIN
assets/svalbard_satellite.jpg


BIN
assets/treatment_plant.jpg


BIN
assets/uk_satellite.jpg


BIN
assets/volcano_satellite.jpg