Depth Based View Synthesis Using Graph Cuts for 3DTV

Anh Tu Tran, Koichi Harada

.
Graduate School of Engineering, Hiroshima University, Higashihiroshima, Japan..

DOI: 10.4236/jsip.2013.43041                

Abstract

In three-dimensional television (3DTV), an interactive free viewpoint selection application has received more attention so far. This paper presents a novel method that synthesizes a free-viewpoint based on multiple textures and depth maps in multi-view camera configuration. This method solves the cracks and holes problem due to sampling rate by performing an inverse warping to retrieve texture images. This step allows a simple and accurate re-sampling of synthetic pixels. To enforce the spatial consistency of color and remove the pixels wrapped incorrectly because of inaccuracy depth maps, we propose some processing steps. The warped depth and warped texture images are used to classify pixels as stable, unstable and disoccluded pixels. The stable pixels are used to create an initial new view by weighted interpolation. To refine the new view, Graph cuts are used to select the best candidates for each unstable pixel. Finally, the remaining disoccluded regions are filled by our inpainting method based on depth information and texture neighboring pixel values. Our experiment on several multi-view data sets is encouraging in both subjective and objective results. Furthermore, our proposal can flexibly use more than two views in multi-view system to create a new view with higher quality.

Keywords

View Synthesis; Depth Image Based Rendering (DIBR); Free-Viewpoint TV; Graph Cuts

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	M. Tanimoto, “Overview of FTV (Free-Viewpoint Television),” Proceedings of the 2009 IEEE International Conference on Multimedia and Expo, New York, 28 June-3 July 2009, pp. 1552-1553. doi:10.1109/ICME.2009.5202803
[2]	C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder and R. Szeliski, “High-Quality Video View Interpolation Using a Layered Representation,” ACM Transactions on Graphics, Vol. 23, No. 3, 2004, pp. 600-608. doi:10.1145/1015706.1015766
[3]	K. Pulli, M. Cohen, T. Duchamp, H. Hoppe, L. G. Shapiro and W. Stuetzle, “View-Base Rendering: Visualizing Real Objects from Scanned Range and Color Data,” Proceedings of the Eurographics Workshop on Rendering Techniques‘97, St. Etienne, 16-18 June 1997, pp. 23-24.
[4]	K. Muller, K. Dix, P. Merkle, P. Kauff and T. Wiegand, “Intermediate View Interpolation Based on Multiview Video plus Depth for Advanced 3D Video Systems,” 15th IEEE International Conference on Image Processing (ICIP), San Diego, 12-15 October 2008, pp. 2448-2451
[5]	Y. Mori, N. Fukushima, T. Yendo, T. Fujii and M. Tanimoto, “View Generation with 3D Warping Using Depth Information for FTV,” Signal Processing-Image Communication, Vol. 24, No. 1-2, 2009, pp. 65-72. doi:10.1016/j.image.2008.10.013
[6]	A. C. Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” Journal of Graphics Tools, Vol. 9, No. 1, 2004, pp. 25-36. doi:10.1080/10867651.2004.10487596
[7]	S. Zinger, L. Do and P. H. N. de With, “Free-Viewpoint Depth Image Based Rendering,” Journal of Visual Communication and Image Representation, Vol. 21, No. 5-6, 2010, pp. 533-541. doi:10.1016/j.jvcir.2010.01.004
[8]	K.-J. Oh, S. Yea and Y.-S. Ho, “Hole Filling Method Using Depth Based In-Painting for View Synthesis in Free Viewpoint Television and 3-D Video,” Picture Coding Symposium, Chicago, 6-8 May 2009, pp. 1-4.
[9]	V. Kolmogorov and R. Zabih, “What Energy Functions Can Be Minimizedvia Graph Cuts?” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 2, 2004, pp. 147-159. doi:10.1109/TPAMI.2004.1262177
[10]	Y. Boykov, O. Veksler and R. Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 11, 2001, pp. 1222-1239. doi:10.1109/34.969114
[11]	Y. Boykov and V. Kolmogorov, “An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 9, 2004, pp. 1124-1137. doi:10.1109/TPAMI.2004.60
[12]	Z. Wang, A. C. Bovik and H. R. Sheikh, “Image Quality Assessment: From Error Measurement to Structural Similarity,” IEEE Transactions on Image Processing, Vol. 13, No. 4, 2004, pp. 600-612. doi:10.1109/TIP.2003.819861
[13]	S. M. Rhee, Y. J. Yoon, I. K. Shin, Y. G. Kim, Y. J. Choi and S. M. Choi, “Stereo Image Synthesis by View Morphing with Stereo Consistency,” Applied Mathematics & Information Sciences, Vol. 6, 2012, pp. 195-200.
[14]	M. Solh and G. AlRegib, “Hierarchical Hole-Filling for Depth-Based View Synthesis in FTV and 3D Video,” IEEE Journal of Selected Topics in Signal Processing, Vol. 6, No. 5, 2012, pp. 495-504. doi:10.1109/JSTSP.2012.2204723