Divide-and-conquer Method with Refined Trimap for High-resolution Image Matting

박현우 서울대학교 대학원 2018 국내석사

High-resolution image matting requires a substantial memory capacity. Therefore, it is very difficult to perform a high-resolution image matting, e.g. KNN image matting, in a common computing environment. This dissertation presents a simple yet novel methodology to solve a memory shortage issue involved in a high-resolution image matting with application of the divide-and-conquer method and a refined trimap. We use a refined trimap to minimize the potential quality degradation in the implementation of the divide-and-conquer method. A confidence map is computed by comparing the original input image and down-sampled image with lower resolution, and the confidence map is used to judge the reliability of the multigrid method in refining the trimap. The original input image is divided into fragments and individual alpha matte is computed with the refined trimap (fragment-level matting); each fragment includes a margin overlapping with nearby fragment(s). Then, we merge the fragments in consideration of the overlapping margin to make sure the merged image does not suffer from discontinuity among fragments. This is the refined D-C method dealing with high-resolution image matting regardless of matting method. We propose this fast and versatile method for high-resolution image matting. This method protects users from potential memory shortage while obtaining a reasonable-quality alpha matte from a high-resolution image.

A framework for video object extraction, deformation, and interactive authoring

이선영 Graduate School, Yonsei University 2012 국내박사

In this dissertation, we introduce several user interactive video editing systems. First we present a video matting system for extracting an object from a video. The state-of-the-art matting algorithm for still images is significantly advanced in terms of speed and quality. However, we cannot directly extend existing matting systems to video, because it would require considerable manual efforts; moreover, it would lead to artifacts due to the temporal incoherence problem. Thus, we present a temporally coherent approach for video matting. Our system extends an existing matting algorithm from 2D to 3D by considering several frames simultaneously rather than frame-by-frame. This method is better than previous techniques in terms of consistency and preservation of temporal coherence. Second, we present a system for transforming the motion and shape of a video object into cartoon-style animation. We apply the cartoon principles that are mainly employed in traditional 2D animation, such as squash-and-stretch, anticipation effects, and follow-and-through, and etc. Furthermore, this method can generate a variety of effects with minimal user input. Finally, we propose a semi-automatic system for adding word balloons to a video us-ing caption information for a movie or TV drama. Previous caption systems used fixed position; hence, viewers would experience a problem during disturbances in the line of sight, and they could not focus on the video content. Our system computes the appro-priate position for word balloons by detecting the faces of characters. For this purpose, we develop a new user authoring tool that minimizes the manual effort required for au-tomatic face and voice verification. Accordingly, we show that our system optimizes the overlaying of word balloons in videos. This dissertation proposes various video editing techniques that can create intuitive and simple tools for non-professional users.

백그라운드 매팅과 피쳐 상관 분석을 통한 유기물 검출 시스템

정민구 中央大學校 尖端映像大學院 2023 국내석사

With the recent spread of CCTV, Abandoned Object detection research is being conducted to detect dangerous objects such as illegal dumping and explosives. Existing abandoned object detection studies have detected abandoned objects using Background Subtraction or deep learning. However, Detection Error occurs due to such as noise, light changes, and similar objects in CCTV. This paper proposes effective abandoned object detection by reducing noise in CCTV environments and lowering detection errors due to similar objects using light change detection and Background Matting algorithms. The main system description of this paper is as follows. i) Pre-Processing ii) Abandoned Object Recognition(AOR) system iii) Abandoned Object Detection by Feature Correlation(AODFC). The proposed system reduces the noise of CCTV and effectively detects only Abandoned objects by removing background matting and dynamic objects. In addition, a more accurate abandoned object detection system is proposed by judging detection errors through feature Correlation. 최근 CCTV가 확산됨에 따라 불법 투기, 폭발물과 같은 위험 물체 검지를 하기 위한 유기물 검지연구가 진행되고 있다. 기존 유기물 검지 연구들은 Background Subtraction이나 딥러닝을 활용하여 유기물을 검지하였다. 하지만 CCTV의 노이즈, 빛 변화, 유기물과 같은 유사 객체로 인한 오 검출이 발생한다. 본 논문에서는 CCTV환경에서 노이즈를 줄이고 빛 변화 감지, Background Matting알고리즘을 사용하여 유사 객체로 인한 오검출을 낮추어 효과적인 유기물 검출을 제안한다. 본 논문의 주요 시스템 설명은 다음과 같다. i) Pre-Processing ii) Abandoned Object Recognition(AOR) system iii) Abandoned Object Detection by Feature Correlation (AODFC). 제안된 시스템은 CCTV의 노이즈를 개선하고, 백그라운드 매팅과 동적 객체가 많은 부분을 제거하여 유기물만을 효과적으로 검출한다. 또한 피쳐 상관 분석을 통하여 오검출을 판단하여 보다 정확한 유기물 검출 시스템을 제안한다.

RGB벡터와 DCT 분석을 활용한 실시간 물체 분류 알고리즘

신현학 경희대학교 일반대학원 2016 국내석사

본 논문에서는 고정된 카메라에서 초기 프레임을 참조하여 현재 프레임에 새롭게 유입된 물체의 실시간 분류 방법을 제안한다. 제안한 알고리즘의 실시간 분류 처리를 위하여 복잡도를 최소화 하였다. 먼저 전경과 배경을 구분하는 이진 분류 영상을 추출하기 위하여 DCT(Discrete Cosine Transform) 기법을 사용한다. DCT는 기존 공간영역에서 Texture를 분석하는 방식보다 더욱 정확하게 Texture를 분석할 수 있다. 이는 주파수 영역에서 Texture 특징 분석이 더욱 용이하고 각 요소 안에 intensity와 texture 정보를 종합적으로 고려할 수 있기 때문이다. 또한 DCT 계산 복잡도를 최소화하기 위하여 DCT 수행 전에 색 정보를 이용하여 미리 분류 영역을 분석함으로써 처리 효율을 극대화 하였다. 마지막으로 생성된 이진 분류 영상을 자연스럽게 matting하기 위하여 Guided 필터 사용을 제안한다. Guided 필터는 guidance 정보를 통해 입력 영상을 전반적으로 개선할 수 있지만 intensity가 평탄한 영역 등에서 그 한계를 보여주므로 본 논문에서는 Guided 필터의 단점을 개선하는 방법을 추가적으로 제안한다.

Multi-Person Image Matting by Using Pose-Estimation

JIN HONGYU 고려대학교 대학원 2023 국내석사

In this paper, we propose a transformer-based model for multi-person human image matting model using pose estimation, which uses information from human pose estimation to guide a transformer-based matting network to improve its performance in human image matting task. Our method takes a matting encoder and a multi-person human pose encoder to extract features from input image, then uses a fusion block to combine and merge these features, finally these merged features will be sent to ResNet decoder to be transformed into matting alpha map. Since most of existing image matting datasets are not focused on real world multi-person human matting, we build a combined human image dataset based on existing datasets and trained our model on this new dataset. Experiments shows that our method achieves high performance on multi-person human image matting task, especially on challenging cases.

3D Depth Reconstruction from Focal Stack and Depth Refinement

마지강 서울대학교 대학원 2021 국내박사

Three-dimensional (3D) depth recovery from two-dimensional images is a fundamental and challenging objective in computer vision, and is one of the most important prerequisites for many applications such as 3D measurement, robot location and navigation, self-driving, and so on. Depth-from-focus (DFF) is one of the important methods to reconstruct a 3D depth in the use of focus information. Reconstructing a 3D depth from texture-less regions is a typical issue associated with the conventional DFF. Further more, it is difficult for the conventional DFF reconstruction techniques to preserve depth edges and fine details while maintaining spatial consistency. In this dissertation, we address these problems and propose an DFF depth recovery framework which is robust over texture-less regions, and can reconstruct a depth image with clear edges and fine details. The depth recovery framework proposed in this dissertation is composed of two processes: depth reconstruction and depth refinement. To recovery an accurate 3D depth, We first formulate the depth reconstruction as a maximum a posterior (MAP) estimation problem with the inclusion of matting Laplacian prior. The nonlocal principle is adopted during the construction stage of the matting Laplacian matrix to preserve depth edges and fine details. Additionally, a depth variance based confidence measure with the combination of the reliability measure of focus measure is proposed to maintain the spatial smoothness, such that the smooth depth regions in initial depth could have high confidence value and the reconstructed depth could be more derived from the initial depth. As the nonlocal principle breaks the spatial consistency, the reconstructed depth image is spatially inconsistent. Meanwhile, it suffers from texture-copy artifacts. To smooth the noise and suppress the texture-copy artifacts introduced in the reconstructed depth image, we propose a closed-form edge-preserving depth refinement algorithm that formulates the depth refinement as a MAP estimation problem using Markov random fields (MRFs). With the incorporation of pre-estimated depth edges and mutual structure information into our energy function and the specially designed smoothness weight, the proposed refinement method can effectively suppress noise and texture-copy artifacts while preserving depth edges. Additionally, with the construction of undirected weighted graph representing the energy function, a closed-form solution is obtained by using the Laplacian matrix corresponding to the graph. The proposed framework presents a novel method of 3D depth recovery from a focal stack. The proposed algorithm shows the superiority in depth recovery over texture-less regions owing to the effective variance based confidence level computation and the matting Laplacian prior. Additionally, this proposed reconstruction method can obtain a depth image with clear edges and fine details due to the adoption of nonlocal principle in the construct]ion of matting Laplacian matrix. The proposed closed-form depth refinement approach shows that the ability in noise removal while preserving object structure with the usage of common edges. Additionally, it is able to effectively suppress texture-copy artifacts by utilizing mutual structure information. The proposed depth refinement provides a general idea for edge-preserving image smoothing, especially for depth related refinement such as stereo vision. Both quantitative and qualitative experimental results show the supremacy of the proposed method in terms of robustness in texture-less regions, accuracy, and ability to preserve object structure while maintaining spatial smoothness.

Real object insertion in single image using optimized sampling-based matting

조선영 Graduate School, Yonsei University 2009 국내석사

Real Object Insertion in Single Image using Optimized Sampling-based Matting

조선영 연세대학교 대학원 2009 국내석사

본 논문에서는 최적화된 샘플링 기반 매팅을 이용한 단일 영상에서의 실사 객체 삽입 방법을 제안한다. 최근 인터넷 및 블로그의 발달로, 사람들은 사진을 직접 조작하거나 편집하여 올리는 일이 많아졌다. 영상을 편집하기 위한 여러가지 방법들 중 객체 삽입은 중요한 수단이 된다. 객체 삽입은 일반적으로 스테레오 영상으로부터 얻어진 3차원 기하학 정보를 이용하여 수행된다. 이러한 멀티 뷰 방법은 정확한 크기 추정을 통해 좋은 결과를 준다. 그러나 스테레오 영상을 획득하기 어려운 데이터가 존재하고, 두 스테레오 영상간의 매칭정보도 찾기 어렵다는 단점이 존재한다. 따라서 최근에는 단일영상으로부터 3차원 기하학 정보를 추출하는 것에 관한 연구가 많이 진행되고 있다. 단일영상을 이용하는 경우, 데이터 획득과 3차원 기하학 정보 추출이 쉽다는 장점이 있다. 본 논문에서는 참조 객체 및 객체의 실제 크기 정보를 이용하여 단일영상에서 자동으로 객체의 크기를 추정하는 방법을 제안한다.사실적이고 자연스러운 객체 삽입을 위해서는 영상으로부터 전경 객체를 추출해 냄으로써 삽입 객체의 테두리 부분을 자연스럽게 합성하기 위한 매팅 기술이 필요하다. 기존의 매팅 기법들은 불확실 영역내의 각 픽셀에서의 이웃 픽셀들을 샘플링하여 알파값을 추정한다. 그러나 이러한 매팅 기법들은 부적절한 이웃 픽셀들을 샘플링함으로써, 잘못된 알파값 추정이 가능하다. 또한, 복잡한 패턴의 영상에 대해서는 정확한 알파값 추정에 실패한다. 이러한 단점을 보완하기 위해, 본 논문에서는 세그멘테이션 정보를 이용하여 이웃 픽셀을 효율적으로 샘플링하는 방법을 제안한다. 제안하는 방법은 자동으로 초기의 대략적으로 주어진 트라이맵을 자세하게 바꾸고, 낮은 신뢰도 값을 갖는 이웃 픽셀들을 제거함으로써 최적의 픽셀들을 샘플링한다. 이러한 트라이맵 및 픽셀 샘플링 최적화 과정은 더 정확한 알파맷을 생성함으로써, 자연스럽게 객체를 삽입하게 한다.실험을 통해, 우리는 객체의 높이 추정이 잘 된 예들을 보인다. 또한, 제안하는 매팅 기법과 기존의 방법들과의 비교를 통해, 제안하는 방법이 기존의 방법들보다 더 정확한 알파맷을 생성함을 보인다. 우리는 객체 높이 추정과 매팅 기법을 이용하여 사실적이고 자연스럽게 객체가 삽입된 것을 보인다. In this thesis, a real object insertion method using optimized sampling-based matting in single image is proposed. As internet and blog have development, many people have uploaded photographs which are manipulated and edited. Object insertion is an important way among many editing methods to manipulate images. Object insertion is generally performed by using 3D geometric information obtained from stereo images. These methods give the good results by accurate height estimation. However, there are data which are difficult to obtain stereo images and matching information. Therefore, techniques for extracting a 3D geometric information from single image have been studied. Single-view metrology techniques have some advantages, for example, it is easy to acquire single-view images and extract 3D geometric information. Based on these advantages, we propose the simple object height estimation algorithm in single image.The matting technique is required to naturally composite an insertion object with background image. Image matting is the process of extracting a foreground element from an image. Previous matting techniques estimate an alpha by sampling the neighborhood pixels of each pixel in unknown region. However, these methods are possible to estimate the erroneous alpha by sampling the improper neighborhood pixels. Also, it is fail to estimate accurate alpha for images of complex patterns. To make up for these drawbacks in previous methods, this paper propose the efficient sampling method of neighborhood pixels using segmentation information. Our approach automatically changes initial naive trimap into detailed trimap, and it uses optimal samples by eliminating pixels with low confidence value. It eventually estimates accurate mattes using the proposed matting method.In our experiments, we present good results of the object height estimation. We also show that proposed matting method can estimate more accurate mattes by experimental results compared with previous approaches. By using height estimation and matting technique, we present real and natural results of object insertion.

TV 혼합현실을 위한 RGB segmentation 최적화 및 분산처리 방법

정영환 고려대학교 컴퓨터정보통신대학원 2020 국내석사

모바일 디바이스와 스마트 글래스의 확산/보급으로 다양한 AR/VR/MR(혼합현실) 어플리케이션이 출시 되었으나, 사용자들이 일반적으로 가정에서 많이 사용하는 대형 화면인 TV에서 이용할 수 있는 AR/VR/MR 어플리케이션은 거의 없는 실정이다. TV 디바이스에서 이러한 오버레이(Overlay) 기법을 이용하려면 별도의 카메라나 CPU가 필요한데, 이를 이용하기 위해 사용자들이 별도의 투자를 하기 어렵기 때문이다. 본 논문에서는 이러한 상황을 디바이스의 분산된 영상처리로 극복하여 누구나 쉽게 고차원의 미디어 서비스를 사용하기 위한 방안을 제시하고자 한다. 사용자에게 가장 친숙한 모바일 디바이스와, 디지털 TV신호를 수신하기 위한 셋톱박스를 연동하여 다양한 AR/VR/MR 서비스를 이용할 수 있는 방법을 제안하고 실제 시스템 구현과 실험을 통해 제안된 방법의 품질 및 실시간성 향상을 검증한다. 특히, TV-모바일 연동 환경에서의 MR을 구현하기 위해 모바일에서 촬영한 영상을 실시간으로 사용자와 배경으로 분리하는 segmentation과 그 과정에서 정확히 분리되지 않는 에러 부분의 정확도를 높이기 위한 여러 방법들에 대한 연구를 기술하였으며, 전송 지연을 최소화하기 위해 생성한 알파맵을 WebM등의 알파 채널의 코덱이 없는 legacy 셋톱박스 장비에서 사용할 수 있도록 전송하는 방법에 대해 기술하였다. 그리고 셋톱박스에서 WebGL을 이용하여 오버레이 형태로 AR서비스를 제공하는 방법에 대해 제시하였다.

A Photograph Reconstruction Method Based on Parameter-Free Spectral Matting for Better Composition

리아즈, 시드라 조선대학교 대학원 2013 국내석사

In digital photography, composition rules are essential for capturing highly aesthetic photographs. Aesthetic images create a response of visual appreciation to the viewers. Rule of thirds (RoT) is the most important and basic rule accepted by photographers for taking aesthetically pleasing shots. The important object is placed along the thirds-lines or intersections of thirds lines. In many cases, the aesthetic quality of the photograph taken from a traditional camera is improved if the photograph is produced to respects the rule of thirds. In this thesis, a novel computational approach, "Retarget Object for Implementation of Rule-of-Thirds'', (ROI-RT) is presented. The technique automatically improves the composition of photographs to follow the composition rule, rule-of-thirds (RoT). For achieving the said task, the main adopted steps are background-foreground estimation, main object segmentation, features extraction, background best patch detection, texture synthesis for occluded background, and retargeting the main object on synthesized background to reproduce a photograph which respect composition rule RoT, in photography. Experimental results show an overall acceptable performance of 94.30% for our method. A comparison with existing techniques shows that ROI-RT performs better in terms of visually aesthetic photographs.