Interestingly, this is effectively an extreme version of solving the colorization of black & white photos problem. I wonder what the results would be if you just threw some black and white photos into the model.