Automatic Image Restoration

Back to my research projects, you also can find the paper of this algorithm from my research projects page.

• Paper and Data
• Motivation
• Diffusion based image restoration
• Texture synthesis based image restoration
• Image Restoration using Multiresolution Texture Synthesis and Image Inpainting
• Examples
• For further understanding : I can write this paper because there are these works.
• FAQ

Paper and Data

• Hitoshi Yamauchi, Jörg Haber and Hans-Peter Seidel, ``Image Restoration using Multiresolution Texture Synthesis and Image Inpainting,'' Proc. Computer Graphics International (CGI) 2003, 2003, pp.120-125, 9-11 July, Tokyo, Japan, Publications' page (Slides are also here.)
• data of the paper.
  You are free to use these data for academic purpose as long as you cite the origin. Such information is in README file in the archive.

Motivation

• Repairing damaged images
  • scratches on pictures or old films
• Fill in missing part of images
  • Images synthesized by IBR, etc.
• Delete unwanted objects on an image
  • subtitles, logos, microphones, ...
• When damaged pixels are indicated by a certain way, I want to repair such region automatically.

Diffusion works well in narrow pixels

When the defect region is only narrow pixels, diffusion (solving partial differential equation) works very well.

• Left Photo from: Image Inpainting, M. Bertalmio, et al., SIGGRAPH 2000.)
• Right Photo by Yamauchi Hitoshi, AG4 volley ball.

Original Image:

Result of Fast Digital Image Inpainting (Oliveira, Manuel M., Brian Bowen, Richard McKenna and Yu-Sung Chang, ``Fast Digital Image Inpainting,'' Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain. September 3-5, 2001, pp. 261-266.):

Result of Image Inpainting (Marcelo Bertalmio and Guillermo Sapiro and Vincent Caselles and Coloma Ballester, ``Image inpainting,'' SIGGRAPH 2000, pp. 417-424):

Large region is problematic.

The original brick texture comes from (VisTex data set Copyright (C) 1995 MIT) http://vismod.media.mit.edu/vismod/imagery/VisionTexture/vistex.html

From left to right : Input Image, Fast Digital Image Inpainting. Notice: This result is only isotropic diffusion.

Texture synthesis works well on homogeneous image

From left to right : Input Image, Texture synthesis.

Texture synthesis works hard on non-homogeneous image

From left to right : Input Image, Texture synthesis.

Image Restoration using Multiresolution Texture Synthesis and Image Inpainting with frequency domain decomposition algorithm (this paper)

From left to right : Input Image, Our method on homogeneous image.

From left to right : Input Image, Our method on non-homogeneous image.

Other examples

Posters: Left input, Right result.

Tables: Left input, Right result.

Ropeway: Left original, Middle masked input, Right result. (Picture: Copyright (C) 2002 Goshima Kazuhiro)

For further understanding

I can write this paper because there are these works.

I tried to complete these links, but it is not possible, as you know. Then, if you know more other related links, please inform me.

My short comments are just memorandums for me.

• David D. Garber : This paper is amazing.
• Vision, Image/Texture analysis
  • Edward H. Adelson, Multiresolution Spline
  • Eero Simoncelli, Steerable pyramid
  • Luis Alvarez
  • Yingnian Wu, Dynamic Textures
  • David Heeger
• Statistical texture analysis/synthesis
  • Jeremy S. de Bonet, Multiresolution Texture Synthesis
  • Kris Popat, Cluster-based probability model
  • Dani Lischinski, Texture Mixing using Statistical Learning
  • Anil C. Kokaram, Detecting and Filling pixels for image sequences
  • Liza Levina
  • Javier Portilla, Using Joint Statistics of Complex Wavelet Coefficients
• Texture synthesis
  • 2D oriented (not limited)
    • Alyosha Efros, wichtig, Non-parametric sampling
    • Rupert Paget, ``Texture synthesis via a noncausal nonparametric multiscale Markov random field''
    • Aaron Hertzmann, Image Analogies. When I saw the texture by numbers, I started to think about this area., Texture and Shape Synthesis on Surfaces
    • Paul Harrison, Resynthesizer, Entropy base
    • Michael Ashikhmin, Natural texture
    • Real-Time Texture Synthesis By Patch-Based Sampling, Baining Guo et al.., Chaos Mosaic
    • Video Textures
    • Cyril.Soler, Hierarchical Pattern Mapping
    • Andrew Nealen, Marc Alexa, Hybrid Texture Synthesis
    • Steve Zelinka, Jump map
    • Ryota Hashimoto, Tomoyuki Nishita, Extend {Image Analogy|Coherent match} for video
    • Iddo Drori, Daniel Cohen-Or, Fragment-Based Image Completion
    • Michael F. Cohen, Wang Tile
    • Homan Igehy, texture replacement
  • 3D oriented.
    • Li-Yi Wei, Texture Analysis and Synthesis. Also you can find synthesis on surface here.
    • Greg Turk, Texture Synthesis on Surfaces
    • Jingdan Zhang et al. (Baining Guo) Synthesis of Progressively-Variant Textures on Arbitrary Surfaces.
      Use texon as a feature, resample with rotation and transition vector field.
• Diffusion based image restoration
  • Marcelo bertalmio, Image inpainting
  • Manuel Oliveira
  • Selim Esedoglu
  • Joachim Weickert, Differential Equations in Image Processing
  • Tony Chan, Imagers : UCLA Applied Math Dep, Image Processing Research group, Mathematical model
  • P. P'erez, M. Gangnet, and A. Blake, Poisson image editing
• Combination or other image manipulations
  • Fr'edo Durand, Image-Based Modeling and Photo Editing
  • Robotics Institute: Takeo Kanade
  • Leo Jiaya Jia, Tensor voting.

FAQ

• Are you interested in making a plugin of this method for an image manipulation software?
  
  

  Yes, I have. I am interested in making a plugin of this method for Adobe PhotoShop or gimp. However:

  • I have no knowledge about to make a PlugIn for both software.
  • I do not know Adobe or gimp society have an interest for this, or not.

  So, currently, I have no plan to make the plugin. My code is just an application, not plugin now. If I have enough lucky, like a company (ex. Adobe, Ghibli, or any other digital image processing company...) hired me for a while, like two or three months?, then, I may do it. But I do not know. It might be quite improbably.

  2004-4-28(Wed): For the students of Saarland university and IMPRS : I've opened a position for Diplomarbeiten, Fopras, HiWi of this ideas in Saarland university and MPII.
  Ich habe eine Stelle für Diplomarbeiten, Fopras, HiWi der Idee anbieten. Bitte schau unsere Stelle Seiten.