GPU-accelerated integral imaging and full-parallax 3D display using stereo–plenoptic camera system

Hong, Seokmin,Incardona, Nicolò,Inoue, Kotaro,Cho, Myungjin,Saavedra, Genaro,Martinez-Corral, Manuel Elsevier 2019 Optics and lasers in engineering Vol.115 No.-

<P><B>Abstract</B></P> <P>In this paper, we propose a novel approach to produce integral images ready to be displayed onto an integral-imaging monitor. Our main contribution is the use of commercial plenoptic camera, which is arranged in a stereo configuration. Our proposed set-up is able to record the radiance, spatial and angular, information simultaneously in each different stereo position. We illustrate our contribution by composing the point cloud from a pair of captured plenoptic images, and generate an integral image from the properly registered 3D information. We have exploited the graphics processing unit (GPU) acceleration in order to enhance the integral-image computation speed and efficiency. We present our approach with imaging experiments that demonstrate the improved quality of integral image. After the projection of such integral image onto the proposed monitor, 3D scenes are displayed with full-parallax.</P> <P><B>Highlights</B></P> <P> <UL> <LI> The use of a stereo–plenoptic camera system. </LI> <LI> Novel for the composing sub-aperture image array from captured plenoptic image and calculate dense depth map. </LI> <LI> Compose a point cloud and registration between a pair of 3D data. </LI> <LI> Generate 3D image (integral image) from composed voxel in virtual 3D space. </LI> <LI> Display 3D images into integral imaging monitor with great parallax and viewing angle. </LI> </UL> </P>

집적 영상 현미경과 집적 영상 디스플레이를 이용한 미세시료의 3차원 영상 재생

임영태,박재형,권기철,김남,Lim, Young-Tae,Park, Jae-Hyeung,Kwon, Ki-Chul,Kim, Nam 한국통신학회 2009 韓國通信學會論文誌 Vol.34 No.11b

집적 영상 현미경에서 획득한 요소 영상을 이용하여 집적 영상 디스플레이에서 재생하였다. 일반화된 서로 다른 두 렌즈 어레이를 이용하는 집적 영상의 획득과 재생의 관계를 이용하여 집적 영상 현미경 및 디스플레이 시스템에 적용하였다. 집적 영상 현미경에서 획득한 요소 영상을 재생하기 위해서 스케일링이 적용된 요소 영상이 필요하다. 집적 영상을 재생할 때 요소 영상의 정보 손실을 최소화하는 스케일링 계수와 요소 영상의 왜곡을 최소화하는 스케일링 계수를 선택해야 한다. 본 연구에서는 $125{\mu}m$ 피치 사이즈의 마이크로 렌즈 어레이를 가지는 집적 영상 현미경으로 시료를 촬영하고 1mm 피치 사이즈의 렌즈 어레이를 가지는 집적 영상 디스플레이로 촬영된 시료를 3차원 영상으로 재생하는 실험을 수행하였다. 이 때, 요소 영상의 스케일링 계수는 집적 영상 현미경에서 획득한 요소 영상의 정보 손실을 최소화 하는 값으로 설정하였다. Microscopic specimen was captured by an integral imaging microscope and displayed as a three-dimensional image by an integral imaging display system. We applied the generalized relationship between pickup and display using two different lens arrays to our integral imaging microscope and display system. In order to display three-dimensional microscopic image, scaling of the captured elemental images is required. We analyzed the effect of the scaling coefficient in terms of the distortion of the displayed three-dimensional image and the loss of the captured elemental images. In our experiment, microscopic specimen is picked up by an integral imaging microscope having $125{\mu}m$ elemental lens pitch and displayed as three-dimensional image by an integral imaging display system having 1mm elemental lens pitch. The scaling coefficient was chosen to minimize the elemental image loss.

Transformations and Their Analysis from a RGBD Image to Elemental Image Array for 3D Integral Imaging and Coding

( Hoon Yoo ) 한국인터넷정보학회 2018 KSII Transactions on Internet and Information Syst Vol.12 No.5

This paper describes transformations between elemental image arrays and a RGBD image for three-dimensional integral imaging and transmitting systems. Two transformations are introduced and analyzed in the proposed method. Normally, a RGBD image is utilized in efficient 3D data transmission although 3D imaging and display is restricted. Thus, a pixel-to-pixel mapping is required to obtain an elemental image array from a RGBD image. However, transformations and their analysis have little attention in computational integral imaging and transmission. Thus, in this paper, we introduce two different mapping methods that are called as the forward and backward mapping methods. Also, two mappings are analyzed and compared in terms of complexity and visual quality. In addition, a special condition, named as the hole-free condition in this paper, is proposed to understand the methods analytically. To verify our analysis, we carry out experiments for test images and the results indicate that the proposed methods and their analysis work in terms of the computational cost and visual quality.

Comparisons of Object Recognition Performance with 3D Photon Counting & Gray Scale Images

이충규,문인규 한국광학회 2010 Current Optics and Photonics Vol.14 No.4

In this paper the object recognition performance of a photon counting integral imaging system is quantitatively compared with that of a conventional gray scale imaging system. For 3D imaging of objects with a small number of photons, the elemental image set of a 3D scene is obtained using the integral imaging set up. We assume that the elemental image detection follows a Poisson distribution. Computational geometrical ray back propagation algorithm and parametric maximum likelihood estimator are applied to the photon counting elemental image set in order to reconstruct the original 3D scene. To evaluate the photon counting object recognition performance, the normalized correlation peaks between the reconstructed 3D scenes are calculated for the varied and fixed total number of photons in the reconstructed sectional image changing the total number of image channels in the integral imaging system. It is quantitatively illustrated that the recognition performance of the photon counting integral imaging system can be similar to that of a conventional gray scale imaging system as the number of image viewing channels in the photon counting integral imaging (PCII) system is increased up to the threshold point. Also, we present experiments to find the threshold point on the total number of image channels in the PCII system which can guarantee a comparable recognition performance with a gray scale imaging system. To the best of our knowledge,this is the first report on comparisons of object recognition performance with 3D photon counting & gray scale images.

메모리 크기에 효율적인 적분영상 하드웨어 설계 연구

이수현(Su-hyun Lee),정용진(Yong-jin Jeong) 대한전자공학회 2014 전자공학회논문지 Vol.51 No.9

적분영상은 입력영상의 픽셀 값을 기준좌표부터 순차적으로 누적하여 만든 영상으로, Haar-like features와 같은 네모난 박스 모양의 필터 연산을 효율적으로 처리하기 위하여 사용된다. 그러나 적분영상은 입력영상보다 3배 이상 많은 메모리를 소모하기 때문에, 메모리 자원이 제한적인 하드웨어 설계 환경에서는 사용이 어렵다. 본 논문에서는 효율적인 메모리 사용을 위한 적분영상 하드웨어 설계 방법을 제안한다. 해당 방법은 적분영상 이외에 세로적분영상과 가로적분영상을 생성하고, 입력영상을 재사용 하는 방법을 사용한다. 그리고 박스 필터의 크기에 따라 modulo 연산을 적용하여 적분영상의 데이터 크기를 줄이는 방법을 함께 적용하였다. 적분 영상 데이터를 읽기 위해 나누어진 영상 데이터를 다시 덧셈해야하는 연산 오버헤드가 발생하지만, 4개의 데이터를 단순히 더하는 연산이므로 병렬처리가 가능한 하드웨어 환경에서는 큰 영향을 미치지 않는다. Xilinx사의 Virtex5-LX330T를 대상으로 실험한 결과 640×480 크기의 8bit gray-scale 입력영상에서 최대 32×32 크기의 필터사용을 기준으로 50%의 적분영상 메모리를 감소시킬 수 있다. The integral image is the sum of input image pixel values. It is mainly used to speed up processing of a box filter operation, such as Haar-like features. However, large memory for integral image data can be an obstacle on an embedded hardware environment with limited memory resources. Therefore, an efficient method to store the integral image is necessary. In this paper, we propose a memory size reduction hardware design for integral image. The hardware design is used two methods. It is the new integral image memory and modulo calculation for reducing integral image data. The new integral image memory has additional calculation overhead, but it is not obstacle in hardware environment that parallel processing is possible. In the Xilinx Virtex5-LX330T targeted experimental result, integral image memory can be reduced by 50% on a 640×480 8-bit gray-scale input image.

Simplified Integral Imaging Pickup Method for Real Objects Using a Depth Camera

Gang Li,권기철,Gwan-Ho Shin,Ji-Seong Jeong,Kwan-Hee Yoo,김남 한국광학회 2012 Current Optics and Photonics Vol.16 No.4

In this paper, we present a novel integral imaging pickup method. We extract each pixel’s actual depth data from a real object’s surface using a depth camera, then generate elemental images based on the depth map. Since the proposed method generates elemental images without a lens array, it has simplified the pickup process and overcome some disadvantages caused by a conventional optical pickup process using a lens array. As a result, we can display a three-dimensional (3D) image in integral imaging. To show the usefulness of the proposed method, an experiment is presented. Though the pickup process has been simplified in the proposed method, the experimental results reveal that it can also display a full motion parallax image the same as the image reconstructed by the conventional method. In addition, if we improve calculation speed, it will be useful in a real-time integral imaging display system.

Visual Quality Enhancement of Three-Dimensional Integral Imaging Reconstruction for Partially Occluded Objects Using Exemplar-Based Image Restoration

Zhang, Miao,Zhong, Zhaolong,Piao, Yongri The Korea Institute of Information and Commucation 2016 Journal of information and communication convergen Vol.14 No.1

In generally, the resolution of reconstructed three-dimensional images can be seriously degraded by undesired occlusions in the integral imaging system, because the undesired information of the occlusion overlap the three-dimensional images to be reconstructed. To solve the problem of the undesired occlusion, we present an exemplar-based image restoration method in integral imaging system. In the proposed method, a minimum spanning tree-based stereo matching method is used to remove the region of undesired occlusions in each elemental image. After that, the removed occlusion region of each elemental images are re-established by using the exemplar-based image restoration method. For further improve the performance of the image restoration, the structure tensor is used to solve the filling error cause by discontinuous structures. Finally, the resolution enhanced three-dimensional images are reconstructed by using the restored elemental images. The preliminary experiments are presented to demonstrate the feasibility of the proposed method.

Design and Implementation of High-Resolution Integral Imaging Display System using Expanded Depth Image

Song, Min-Ho,Lim, Byung-Muk,Ryu, Ga-A,Ha, Jong-Sung,Yoo, Kwan-Hee The Korea Contents Association 2018 International Journal of Contents Vol.14 No.3

For 3D display applications, auto-stereoscopic display methods that can provide 3D images without glasses have been actively developed. This paper is concerned with developing a display system for elemental images of real space using integral imaging. Unlike the conventional method, which reduces a color image to the level as much as a generated depth image does, we have minimized original color image data loss by generating an enlarged depth image with interpolation methods. Our method was efficiently implemented by applying a GPU parallel processing technique with OpenCL to rapidly generate a large amount of elemental image data. We also obtained experimental results for displaying higher quality integral imaging rather than one generated by previous methods.

집적 영상에서 시점 영상 배열 변환에 기반 둔 요소 영상 압축

오은주(Eun-Ju Oh),유훈(Hoon Yoo) 대한전기학회 2020 전기학회논문지 Vol.69 No.1

This paper proposes a compression method for elemental images based on the view image array transform in integral imaging. As three-dimensional images start to get attention, the integral imaging as an three-dimensional imaging method has been studied actively. Elemental images have a large amount of data since they offer lots of different view points than regular images. For this reason, studies on pre-processing and compression for elemental images are required. In this paper, we proposed a compression method for elemental images based on view image array. An experiment is conducted to evaluate our method. The experimental results indicate that proposed method is effective for the elemental image compression.

Visual quality enhancement of integral imaging by using pixel rearrangement technique with convolution operator (CPERTS)

Inoue, Kotaro,Cho, Myungjin Elsevier 2018 Optics and lasers in engineering Vol.111 No.-

<P><B>Abstract</B></P> <P>In this paper, we propose a new visual quality enhancement of a three-dimensional (3D) computational reconstruction algorithm in integral imaging. Integral imaging can record 3D images easily using a lenslet array. However, the elemental images may have low resolution, because each image cannot use full resolution of an image sensor. To solve this problem, a computational reconstruction technique can be used to reconstruct visual-quality-enhanced 3D images from low-resolution elemental images. Our method is based on the pixel of elemental images rearrangement technique (PERT), which can provide enhanced visual quality of the reconstructed 3D image compared with that of conventional computational reconstruction algorithms. However, it has a problem in which the size of 3D scenes is different from the optical reconstruction results. Therefore, in this paper, we propose a solution considering empty spaces between back-projected pixels on the reconstruction planes and enhance the extensibility using the convolution operator. Our experimental results show the enhancement of the visual quality and resolution of the reconstructed 3D images using the point-spread function filter.</P> <P><B>Highlights</B></P> <P> <UL> <LI> Captured multi-view images by integral imaging are low-resolution images since image sensors are divided and used by lenslet arrays. </LI> <LI> Conventional 3D reconstruction techniques cannot provide high-resolution 3D images from low-resolution multi-view images. </LI> <LI> Pixel-to-pixel mapping technique can provide accurate pixel positions on reconstruction planes. </LI> <LI> Our proposed method (CPERTS) reconstructs visual quality enhanced 3D images by considering the diffraction pattern of the virtual pinhole using convolution operator. </LI> </UL> </P>