A Sketch-based 3D Object Retrieval Approach for Augmented Reality Models Using Deep Learning

다국어 초록 (Multilingual Abstract)

Retrieving a 3D model from a 3D database and augmenting the retrieved model in the Augmented Reality system simultaneously became an issue in developing the plausible AR environments in a convenient fashion. It is considered that the sketch-based 3D object retrieval is an intuitive way for searching 3D objects based on human-drawn sketches as query. In this paper, we propose a novel deep learning based approach of retrieving a sketch-based 3D object as for an Augmented Reality Model. For this work, we introduce a new method which uses Sketch CNN, Wasserstein CNN and Wasserstein center loss for retrieving a sketch-based 3D object. Especially, Wasserstein center loss is used for learning the center of each object category and reducing the Wasserstein distance between center and features of the same category. The proposed 3D object retrieval and augmentation consist of three major steps as follows. Firstly, Wasserstein CNN extracts 2D images taken from various directions of 3D object using CNN, and extracts features of 3D data by computing the Wasserstein barycenters of features of each image. Secondly, the features of the sketch are extracted using a separate Sketch CNN. Finally, we adopt sketch-based object matching method to localize the natural marker of the images to register a 3D virtual object in AR system. Using the detected marker, the retrieved 3D virtual object is augmented in AR system automatically. By the experiments, we prove that the proposed method is efficiency for retrieving and augmenting objects.

참고문헌 (Reference)

1 Nicolas Bonneel, "Wasserstein barycentric coordinates" Association for Computing Machinery (ACM) 35 (35): 1-10, 2016

2 L. van der Maaten, "Visualizing highdimensional data using t-SNE" 9 : 2579-2605, 2008

3 He, Xinwei, "Triplet-Center Loss for Multi-View 3D Object Retrieval"

4 Vladimir I Bogachev, "The Monge-Kantorovich problem: achievements, connections, and perspectives" IOP Publishing 67 (67): 785-890, 2012

5 Yossi Rubner, "The Earth Mover’s Distance as a metric for image retrieval" Springer Science and Business Media LLC 40 (40): 99-121, 2000

6 S. Ferradans, "Static and Dynamic Texture Mixing Using Optimal Transport" 137-148, 2013

7 Mathias Eitz, "Sketch-based shape retrieval" Association for Computing Machinery (ACM) 31 (31): 1-10, 2012

8 Fang Wang, "Sketch-based 3D shape retrieval using Convolutional Neural Networks" 2015

9 M. Eitz, "Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors" Institute of Electrical and Electronics Engineers (IEEE) 17 (17): 1624-1636, 2011

10 M. Cuturi, "Sinkhorn distances: Lightspeed computation of optimal transport" 2292-2300, 2013

11 B. Li, "Shrec’13 track: Large scale sketchbased 3D shape retrieval" 89-96, 2013

12 T. Furuya, "Ranking on cross-domain manifold for sketch-based 3D model retrieval" 274-281, 2013

13 Y. Wen, "Lecture Notes in Computer Science" 499-515, 2016

14 J. Xie, "Learning Barycentric Representations of 3D Shapes for Sketch-Based 3D Shape Retrieval" 2017

15 Jean-David Benamou, "Iterative Bregman Projections for Regularized Transportation Problems" Society for Industrial & Applied Mathematics (SIAM) 37 (37): A1111-A1138, 2015

16 A. Rolet, "Fast dictionary learning with a smoothed wasserstein loss" 630-638, 2016

17 F. Schroff, "FaceNet : A unified embedding for face recognition and clustering" 2015

18 B. Li, "Extended large scale sketch-based 3D shape retrieval" 121-130, 2014

19 Loris Nanni, "Ensemble of shape descriptors for shape retrieval and classification" Inderscience Publishers 6 (6): 136-156, 2014

20 R. Hadsell, "Dimensionality Reduction by Learning an Invariant Mapping" 2 : 2006

21 Richard Sinkhorn, "Diagonal Equivalence to Matrices with Prescribed Row and Column Sums" JSTOR 74 (74): 402-, 1967

22 K. He, "Deep Residual Learning for Image Recognition" 2016

23 K.V. Shriram, "An intelligent system of content-based image retrieval for crime investigation" Inderscience Publishers 7 (7): 264-279, 2015

24 Bo Li, "A comparison of methods for sketch-based 3D shape retrieval" Elsevier BV 119 : 57-80, 2014

25 Pieter-Tjerk de Boer, "A Tutorial on the Cross-Entropy Method" Springer Science and Business Media LLC 134 (134): 19-67, 2005