|
@@ -0,0 +1,275 @@
|
|
1
|
+---
|
|
2
|
+title: Generating Realistic Satellite Imagery with Deep Neural Networks
|
|
3
|
+layout: post
|
|
4
|
+---
|
|
5
|
+
|
|
6
|
+I've been doing a lot of experimenting with [neural-style](https://github.com/jcjohnson/neural-style)
|
|
7
|
+the last month. I think I've discovered a few exciting applications of the
|
|
8
|
+technique that I haven't seen anyone else do yet. The true power of this
|
|
9
|
+algorithm really shines when you can see concrete examples.
|
|
10
|
+
|
|
11
|
+Skip to the **Applications** part of this post to see the outputs from my
|
|
12
|
+experimentation if you are already familiar with DeepDream, Deep Style, and all
|
|
13
|
+the other latest research in generating images with deep neural networks.
|
|
14
|
+
|
|
15
|
+###Background and History###
|
|
16
|
+
|
|
17
|
+On [May 18, 2015 at 2 a.m., Alexander
|
|
18
|
+Mordvintsev](https://medium.com/backchannel/inside-deep-dreams-how-google-made-its-computers-go-crazy-83b9d24e66df#.g4t69y8wy),
|
|
19
|
+an engineer at Google, did something with deep neural networks that no one had
|
|
20
|
+done before. He took a net designed for *recognizing* objects in images and used
|
|
21
|
+it to *generate* objects in images. In a sense, he was telling these systems
|
|
22
|
+that mimic the human visual cortex to hallucinate things that weren't really
|
|
23
|
+there. The [results](https://i.imgur.com/6ocuQsZ.jpg) looked remarkably like LSD
|
|
24
|
+trips or what a [schizophrenic person sees on a blank
|
|
25
|
+wall](https://www.reddit.com/r/deepdream/comments/3cewgn/an_artist_suffering_from_schizophrenia_was_told/).
|
|
26
|
+
|
|
27
|
+Mordvintsev's discovery quickly gathered attention at Google once he posted
|
|
28
|
+images from his experimentation on the company's internal network. On June 17,
|
|
29
|
+2015, [Google posted a blog post about the
|
|
30
|
+technique](http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html)
|
|
31
|
+(dubbed "Inceptionism") and how it was useful for opening up the notoriously
|
|
32
|
+black-boxed neural networks using visualizations that researchers could examine.
|
|
33
|
+These machine hallucinations were key for identifying the features of objects
|
|
34
|
+that neural networks used to tell one object from another (like a dog from a
|
|
35
|
+cat). But the post also revealed the [beautiful
|
|
36
|
+results](https://goo.gl/photos/fFcivHZ2CDhqCkZdA) of applying the algorithm
|
|
37
|
+iteratively on it's own outputs and zooming out at each step.
|
|
38
|
+
|
|
39
|
+The internet exploded in response to this post. And once [Google posted the code
|
|
40
|
+for performing the
|
|
41
|
+technique](http://googleresearch.blogspot.com/2015/07/deepdream-code-example-for-visualizing.html?m=1),
|
|
42
|
+people began experimenting and sharing [their fantastic and creepy
|
|
43
|
+images](https://www.reddit.com/r/deepdream) with the world.
|
|
44
|
+
|
|
45
|
+Then, on August, 26, 2015, a paper titled ["A Neural Algorithm of Artistic
|
|
46
|
+Style"](http://arxiv.org/abs/1508.06576) was published. It showed how one could
|
|
47
|
+identify which layers of deep neural networks recognized stylistic information
|
|
48
|
+of an image (and not the content) and then use this stylistic information in
|
|
49
|
+Google's Inceptionism technique to paint other images in the style of any
|
|
50
|
+artist. A [few](https://github.com/jcjohnson/neural-style)
|
|
51
|
+[implementations](https://github.com/kaishengtai/neuralart) of the paper were
|
|
52
|
+put up on Github. This exploded the internet again in a frenzy. This time, the
|
|
53
|
+images produced were less like psychedelic-induced nightmares but more like the
|
|
54
|
+next generation of Instagram filters ([reddit
|
|
55
|
+how-to](https://www.reddit.com/r/deepdream/comments/3jwl76/how_anyone_can_create_deep_style_images/)).
|
|
56
|
+
|
|
57
|
+People began to wonder [what all of this
|
|
58
|
+meant](http://www.hopesandfears.com/hopes/culture/is-this-art/215039-deep-dream-google-art)
|
|
59
|
+to [the future of
|
|
60
|
+art](http://kajsotala.fi/2015/07/deepdream-today-psychedelic-images-tomorrow-unemployed-artists/).
|
|
61
|
+Some of the results produced where [indistinguishable from the style of dead
|
|
62
|
+artists'
|
|
63
|
+works](https://raw.githubusercontent.com/jcjohnson/neural-style/master/examples/outputs/tubingen_starry.png).
|
|
64
|
+Was this a demonstration of creativity in computers or just a neat trick?
|
|
65
|
+
|
|
66
|
+On November, 19, 2015, [another paper](http://arxiv.org/abs/1511.06434) was
|
|
67
|
+released that demonstrated a technique for generating scenes from convolutional
|
|
68
|
+neural nets ([implementation on Github](https://github.com/Newmu/dcgan_code)).
|
|
69
|
+The program could generate random (and very realistic) [bedroom
|
|
70
|
+images](https://github.com/Newmu/dcgan_code/raw/master/images/lsun_bedrooms_five_epoch_samples.png)
|
|
71
|
+from a neural net trained on bedroom images. Amazingly, it could also generate
|
|
72
|
+[the same bedroom from any
|
|
73
|
+angle](https://github.com/Newmu/dcgan_code/blob/master/images/lsun_bedrooms_five_epochs_interps.png).
|
|
74
|
+It could also [produce images of the same procedurally generated face from any
|
|
75
|
+angle](https://github.com/Newmu/dcgan_code/blob/master/images/turn_vector.png).
|
|
76
|
+Theoretically, we could use this technology to create *procedurally generated
|
|
77
|
+game art*.
|
|
78
|
+
|
|
79
|
+The main thing holding this technology back from revolutionizing procedurally
|
|
80
|
+generated video games is that it is not real-time. Using
|
|
81
|
+[neural-style](https://github.com/jcjohnson/neural-style) to apply artistic
|
|
82
|
+style to a 512 by 512 pixel content image could take minutes even on the
|
|
83
|
+top-of-the-line GTX Titan X graphics card. Still, I believe this technology has
|
|
84
|
+a lot of potential for generating game art even if it can't act as a real-time
|
|
85
|
+filter.
|
|
86
|
+
|
|
87
|
+###Applications: Generating Satellite Images for Procedural World Maps###
|
|
88
|
+
|
|
89
|
+I personally know very little machine learning, but I have been able to produce
|
|
90
|
+a lot of interesting results by using the tool provided by
|
|
91
|
+[neural-style](https://github.com/jcjohnson/neural-style).
|
|
92
|
+
|
|
93
|
+Inspired by [Kaelan's procedurally generated world
|
|
94
|
+maps](http://blog.kaelan.org/randomly-generated-world-map/), I wanted to extend
|
|
95
|
+the idea by generating realistic satellite images of the terrain maps. The
|
|
96
|
+procedure is simple: take a [generated terrain map](/assets/kaelan_terrain1.png)
|
|
97
|
+and apply the style of a [real-world satellite image](/assets/uk_satellite.jpg)
|
|
98
|
+on it using neural-style.
|
|
99
|
+
|
|
100
|
+![Output of generated map plus real-world satellite
|
|
101
|
+imagery](/assets/satellite_terrain1_process.png)
|
|
102
|
+
|
|
103
|
+The generated output takes on whatever terrain is in the satellite image. Here
|
|
104
|
+is an output processing one of Kaelan's maps with a [arctic satellite
|
|
105
|
+image](/assets/svalbard_satellite.jpg):
|
|
106
|
+
|
|
107
|
+![Kaelan's terrain map](/assets/kaelan_terrain2.jpg)
|
|
108
|
+![Output of terrain map plus arctic satellite imagery](/assets/satellite_terrain2.png)
|
|
109
|
+
|
|
110
|
+And again, with one of Kaelan's desert maps and a [satellite image of a
|
|
111
|
+desert](/assets/desert_satellite.jpg):
|
|
112
|
+
|
|
113
|
+![Kaelan's desert terrain map](/assets/kaelan_terrain3.jpg)
|
|
114
|
+![Output of terrain map plus desert satellite imagery](/assets/satellite_terrain3.png)
|
|
115
|
+
|
|
116
|
+It even works with [Kaelan's generated hexagon
|
|
117
|
+maps](http://blog.kaelan.org/hexagon-world-map-generation/). Here's an island
|
|
118
|
+hexagon map plus a [satellite image of a volcanic
|
|
119
|
+island](/assets/volcano_satellite.jpg):
|
|
120
|
+
|
|
121
|
+![Kaelan's island hexagon map](/assets/kaelan_hex_terrain.jpg)
|
|
122
|
+![Output of hexagon map plus island satellite
|
|
123
|
+imagery](/assets/satellite_hex_terrain.png)
|
|
124
|
+
|
|
125
|
+This image even produced an interesting three-dimensional effect because of the
|
|
126
|
+volcano in the satellite image.
|
|
127
|
+
|
|
128
|
+By the way, this also works with minecraft maps. Here's a minecraft map I found
|
|
129
|
+on the internet plus a [satellite image from Google
|
|
130
|
+Earth](/assets/river_satellite.png):
|
|
131
|
+
|
|
132
|
+![Minecraft map](/assets/minecraft_map.jpg)
|
|
133
|
+![Output of minecraft map plus river satellite
|
|
134
|
+imagery](/assets/satellite_minecraft_map.png)
|
|
135
|
+
|
|
136
|
+No fancy texture packs or 3-D rendering needed :).
|
|
137
|
+
|
|
138
|
+Here is the Fallout 4 grayscale map plus a
|
|
139
|
+[satellite image of Boston](/assets/boston_aerial.jpg):
|
|
140
|
+
|
|
141
|
+![Fallout 4 grayscale map](/assets/fallout4_map.png)
|
|
142
|
+![Output of Fallout 4 map plus Boston satellite
|
|
143
|
+imagery](/assets/satellite_fallout4_map.png)
|
|
144
|
+
|
|
145
|
+Unfortunately, it puts the built-up dense part of the city in the wrong part of
|
|
146
|
+the geographic area. But, this is understandable since we gave the algorithm no
|
|
147
|
+information on where that is on the map.
|
|
148
|
+
|
|
149
|
+We can also make the generated terrain maps look like old hand-drawn maps using
|
|
150
|
+neural-style. With Kaelan's terrain map as the
|
|
151
|
+content and [the in-game Elder Scrolls IV Oblivion map of
|
|
152
|
+Cyrodiil](/assets/cyrodiil_ingame.jpg) as the style we get this:
|
|
153
|
+
|
|
154
|
+![Kaelan's terrain map](/assets/kaelan_terrain1.png)
|
|
155
|
+![Output of terrain map plus map of Cyrodiil](/assets/cyrodiil_terrain1.png)
|
|
156
|
+
|
|
157
|
+It looks cool, but the water isn't conveyed very clearly (e.g. makes deep water
|
|
158
|
+look like land). Neural-style seems to work better when there is lots of color
|
|
159
|
+in both images.
|
|
160
|
+
|
|
161
|
+Here is the output of the hex terrain plus satellite map above and the Cyrodiil
|
|
162
|
+map which looks a little cleaner:
|
|
163
|
+
|
|
164
|
+![Satellite-like hex terrain map](/assets/satellite_hex_terrain.png)
|
|
165
|
+![Output of hex terrain plus satellite and map of
|
|
166
|
+Cyrodiil](/assets/cyrodiil_satellite_hex_terrain.png)
|
|
167
|
+
|
|
168
|
+I was interested to see what neural-style could generate from random noise, so I
|
|
169
|
+rendered some clouds in GIMP and ran it with a satellite image of [Mexico City
|
|
170
|
+from Google Earth](/assets/mexico_city.jpg) (by the way, I've been getting high
|
|
171
|
+quality Google Earth shots from
|
|
172
|
+[earthview.withgoogle.com](https://earthview.withgoogle.com)).
|
|
173
|
+
|
|
174
|
+![Random clouds](/assets/blurry_clouds.png)
|
|
175
|
+![Output of random clouds and Mexico City](/assets/random_mexico_city.png)
|
|
176
|
+
|
|
177
|
+Not bad for a neural net without a degree in urban planning.
|
|
178
|
+
|
|
179
|
+I also tried generating on random noise with a satellite image of [a water
|
|
180
|
+treatment plant in Peru](/assets/treatment_plant.jpg)
|
|
181
|
+
|
|
182
|
+![Random clouds](/assets/blurry_clouds2.png)
|
|
183
|
+![Output of random clouds and water treatment
|
|
184
|
+plant](/assets/random_treatment_plant.png)
|
|
185
|
+
|
|
186
|
+###Applications: More Fun###
|
|
187
|
+
|
|
188
|
+For fun, here are some other outputs that I liked.
|
|
189
|
+
|
|
190
|
+[My photo of Boston's skyline as the content](/assets/boston_skyline.jpg) and
|
|
191
|
+[Vincent van Gogh's The Starry Night as the style](/assets/starry_night.jpg):
|
|
192
|
+
|
|
193
|
+![Output of Boston skyline and starry night](/assets/starry_boston.png)
|
|
194
|
+
|
|
195
|
+[A photo of me](/assets/standing_forest.jpg) (by Aidan Bevacqua) and [Forrest in
|
|
196
|
+the end of Autumn by Caspar David Friedrich](/assets/forrest_autumn.jpg):
|
|
197
|
+
|
|
198
|
+![Output of me and Forrest in the end of
|
|
199
|
+Autumn](/assets/dead_forest_standing.png)
|
|
200
|
+
|
|
201
|
+[Another photo of me by Aidan](/assets/sitting_forest.jpg) in the same style:
|
|
202
|
+
|
|
203
|
+![Output of me and Forrest in the end of Autumn](/assets/dead_forest_sitting.png)
|
|
204
|
+
|
|
205
|
+[A photo of me on a mountain](/assets/mountain_view.jpg) (by Aidan Bevacqua) and
|
|
206
|
+[pixel art by Paul Robertson](/assets/pixels.png)
|
|
207
|
+
|
|
208
|
+![Output of me on a mountain and pixel art](/assets/mountain_view_pixels.png)
|
|
209
|
+
|
|
210
|
+[A photo of a park in Copenhagen I took](/assets/copenhagen_park.jpg) and a
|
|
211
|
+painting similar in composition, [Avenue of Poplars at Sunset by Vincent van
|
|
212
|
+Gogh](/assets/avenue_poplars.jpg):
|
|
213
|
+
|
|
214
|
+![Output of park in Copenhagen and Avenue of Poplars at
|
|
215
|
+Sunset](/assets/poplars.png)
|
|
216
|
+
|
|
217
|
+[My photo of the Shenandoah National Park](/assets/shenandoah_mountains.jpg) and
|
|
218
|
+[this halo graphic from GMUNK](/assets/halo_ring_mountains.jpg)
|
|
219
|
+([GMUNK](http://www.gmunk.com/filter/Interactive/ORA-Summoners-HALO)):
|
|
220
|
+
|
|
221
|
+![Output of Shenandoah mountains and halo ring
|
|
222
|
+mountains](/assets/halo_shenandoah.png)
|
|
223
|
+
|
|
224
|
+[A photo of me by Aidan](/assets/me.png) and a [stained glass
|
|
225
|
+fractal](/assets/stained_glass.jpg):
|
|
226
|
+
|
|
227
|
+![Output of me and a stained glass fractal](/assets/stained_glass_portrait.png)
|
|
228
|
+
|
|
229
|
+Same photo of me and some [psychedelic art by GMUNK](/assets/pockets.jpg)
|
|
230
|
+
|
|
231
|
+![Output of me and psychedelic art](/assets/pockets_portrait.png)
|
|
232
|
+
|
|
233
|
+[New York City](/assets/nyc.jpg) and [a rainforest](/assets/rainforest.jpg):
|
|
234
|
+
|
|
235
|
+![Output of New York City and a rainforest](/assets/jungle_nyc.png)
|
|
236
|
+
|
|
237
|
+[Kowloon Walled City](/assets/kowloon.jpg) and [a National Geographic
|
|
238
|
+Map](/assets/ngs_map.jpg):
|
|
239
|
+
|
|
240
|
+![Output of Kowloon and NGS map](/assets/kowloon_ngs.png)
|
|
241
|
+
|
|
242
|
+[A photo of me by Aidan](/assets/side_portrait.jpg) and [Head of Lioness by
|
|
243
|
+Theodore Gericault](/assets/head_lioness.jpg):
|
|
244
|
+
|
|
245
|
+![Output of photo of me and ](/assets/lion_portrait.png)
|
|
246
|
+
|
|
247
|
+[Photo I took of a Norwegian forest](/assets/forest_hill.jpg) and [The Mountain
|
|
248
|
+Brook by Albert Bierstadt](/assets/mountain_brook.jpg):
|
|
249
|
+
|
|
250
|
+![Output of Norwegian forest and The Mountain
|
|
251
|
+Brook](/assets/mountain_brook_hill.png)
|
|
252
|
+
|
|
253
|
+###Limitations###
|
|
254
|
+
|
|
255
|
+I don't have infinite money for a GTX Titan X, so I'm stuck with using OpenCL on
|
|
256
|
+my more-than-a-few-generations-old AMD card. It takes about a half-hour to
|
|
257
|
+generate one 512x512 px image in my set-up (which makes the feedback loop for
|
|
258
|
+correcting mistakes *very* long). And sometimes the neural-style refuses to run
|
|
259
|
+on my GPU (I suspect it runs out of VRAM), so I have to run it on my CPU which
|
|
260
|
+takes even longer...
|
|
261
|
+
|
|
262
|
+I am unable to generate bigger images (though
|
|
263
|
+[the author has been able to generate up to 1920x1010
|
|
264
|
+px](https://github.com/jcjohnson/neural-style/issues/36#issuecomment-142994812)).
|
|
265
|
+As the size of the output increases the amount of memory and time to generate
|
|
266
|
+also increases. And, it's not practical to just generate thumbnails to test
|
|
267
|
+parameters, because increasing the image size will probably generate a very
|
|
268
|
+different image since all the other parameters stay the same even though they
|
|
269
|
+are dependent on the image size.
|
|
270
|
+
|
|
271
|
+Some people have had success running these neural nets on GPU spot instances in
|
|
272
|
+AWS. It would be certainly cheaper than buying a new GPU in the short-term.
|
|
273
|
+
|
|
274
|
+So, I have a few more ideas for what to run, but it will take me quite a while
|
|
275
|
+to get through the queue.
|