2D–3D pose consistency-based conditional random fields for 3D human pose estimation

Chang, Ju Yong,Lee, Kyoung Mu Elsevier 2018 Computer vision and image understanding Vol.169 No.-

<P><B>Abstract</B></P> <P>This study considers the 3D human pose estimation problem in a single RGB image by proposing a conditional random field (CRF) model over 2D poses, in which the 3D pose is obtained as a byproduct of the inference process. The unary term of the proposed CRF model is defined based on a powerful heat-map regression network, which has been proposed for 2D human pose estimation. This study also presents a regression network for lifting the 2D pose to 3D pose and proposes the prior term based on the consistency between the estimated 3D pose and the 2D pose. To obtain the approximate solution of the proposed CRF model, the N-best strategy is adopted. The proposed inference algorithm can be viewed as sequential processes of bottom-up generation of 2D and 3D pose proposals from the input 2D image based on deep networks and top-down verification of such proposals by checking their consistencies. To evaluate the proposed method, we use two large-scale datasets: Human3.6M and HumanEva. Experimental results show that the proposed method achieves the state-of-the-art 3D human pose estimation performance.</P> <P><B>Highlights</B></P> <P> <UL> <LI> We propose a new CRF model with a novel 2D–3D pose consistency prior for 3D human pose estimation. </LI> <LI> We propose a simple but powerful 2D-to-3D pose lifting method based on MLP. </LI> <LI> We show the solution of the proposed CRF model can be efficiently found by the N-best strategy. </LI> <LI> Thorough experiments show that the proposed method achieves state-of-the-art 3D human pose estimation performance. </LI> </UL> </P>

Robust 2D human upper-body pose estimation with fully convolutional network

Lee, Seunghee,Koo, Jungmo,Kim, Jinki,Myung, Hyun Techno-Press 2018 Advances in robotics research Vol.2 No.2

With the increasing demand for the development of human pose estimation, such as human-computer interaction and human activity recognition, there have been numerous approaches to detect the 2D poses of people in images more efficiently. Despite many years of human pose estimation research, the estimation of human poses with images remains difficult to produce satisfactory results. In this study, we propose a robust 2D human body pose estimation method using an RGB camera sensor. Our pose estimation method is efficient and cost-effective since the use of RGB camera sensor is economically beneficial compared to more commonly used high-priced sensors. For the estimation of upper-body joint positions, semantic segmentation with a fully convolutional network was exploited. From acquired RGB images, joint heatmaps accurately estimate the coordinates of the location of each joint. The network architecture was designed to learn and detect the locations of joints via the sequential prediction processing method. Our proposed method was tested and validated for efficient estimation of the human upper-body pose. The obtained results reveal the potential of a simple RGB camera sensor for human pose estimation applications.

2.5D human pose estimation for shadow puppet animation

( Shiguang Liu ),( Guoguang Hua ),( Yang Li ) 한국인터넷정보학회 2019 KSII Transactions on Internet and Information Syst Vol.13 No.4

Digital shadow puppet has traditionally relied on expensive motion capture equipments and complex design. In this paper, a low-cost driven technique is presented, that captures human pose estimation data with simple camera from real scenarios, and use them to drive virtual Chinese shadow play in a 2.5D scene. We propose a special method for extracting human pose data for driving virtual Chinese shadow play, which is called 2.5D human pose estimation. Firstly, we use the 3D human pose estimation method to obtain the initial data. In the process of the following transformation, we treat the depth feature as an implicit feature, and map body joints to the range of constraints. We call the obtain pose data as 2.5D pose data. However, the 2.5D pose data can not better control the shadow puppet directly, due to the difference in motion pattern and composition structure between real pose and shadow puppet. To this end, the 2.5D pose data transformation is carried out in the implicit pose mapping space based on self-network and the final 2.5D pose expression data is produced for animating shadow puppets. Experimental results have demonstrated the effectiveness of our new method.

2D Human Pose Estimation based on Object Detection using RGB-D information

( Seohee Park ),( Myunggeun Ji ),( Junchul Chun ) 한국인터넷정보학회 2018 KSII Transactions on Internet and Information Syst Vol.12 No.2

In recent years, video surveillance research has been able to recognize various behaviors of pedestrians and analyze the overall situation of objects by combining image analysis technology and deep learning method. Human Activity Recognition (HAR), which is important issue in video surveillance research, is a field to detect abnormal behavior of pedestrians in CCTV environment. In order to recognize human behavior, it is necessary to detect the human in the image and to estimate the pose from the detected human. In this paper, we propose a novel approach for 2D Human Pose Estimation based on object detection using RGB-D information. By adding depth information to the RGB information that has some limitation in detecting object due to lack of topological information, we can improve the detecting accuracy. Subsequently, the rescaled region of the detected object is applied to ConVol.utional Pose Machines (CPM) which is a sequential prediction structure based on ConVol.utional Neural Network. We utilize CPM to generate belief maps to predict the positions of keypoint representing human body parts and to estimate human pose by detecting 14 key body points. From the experimental results, we can prove that the proposed method detects target objects robustly in occlusion. It is also possible to perform 2D human pose estimation by providing an accurately detected region as an input of the CPM. As for the future work, we will estimate the 3D human pose by mapping the 2D coordinate information on the body part onto the 3D space. Consequently, we can provide useful human behavior information in the research of HAR.

단일 이미지에 기반을 둔 사람의 포즈 추정에 대한 연구 동향

조정찬 한국차세대컴퓨팅학회 2019 한국차세대컴퓨팅학회 논문지 Vol.15 No.5

With the recent development of deep learning technology, remarkable achievements have been made in many research areas of computer vision. Deep learning has also made dramatic improvement in two-dimensional or three-dimensional human pose estimation based on a single image, and many researchers have been expanding the scope of this problem. The human pose estimation is one of the most important research fields because there are various applications, especially it is a key factor in understanding the behavior, state, and intention of people in image or video analysis. Based on this background, this paper surveys research trends in estimating human poses based on a single image. Because there are various research results for robust and accurate human pose estimation, this paper introduces them in two separated subsections: 2D human pose estimation and 3D human pose estimation. Moreover, this paper summarizes famous data sets used in this field and introduces various studies which utilize human poses to solve their own problem. 최근 딥러닝 기술이 발전함에 따라 많은 컴퓨터 비전 연구 분야에서 주목할 만한 성과들이 지속적으로 나오고 있다. 단일 이미지를 기반으로 사람의 2차원 및 3차원 포즈를 추정하는 연구에서도 비약적인 성능향상을 보여주고 있으며, 많은 연구자들이 문제의 범위를 확장하며 활발한 연구 활동을 진행하고 있다. 사람의 포즈 추정은 다양한 응용 분야가 존재하고, 특히 이미지나 비디오 분석에서 사람의 포즈는 행동 및 상태, 의도 파악을 위한 핵심 요소가 되기 때문에 상당히 중요한 연구 분야이다. 이러한 배경에 따라 본 논문은 단일 이미지를 기반으로 한 사람의 포즈 추정 기술에 대한 연구 동향을 살펴보고자 한다. 강인하고 정확한 문제 해결을 위해 다양한 연구 활동 결과가 존재한다는 점에서 본 논문에서는 사람의 포즈 추정 연구를 2차원 및 3차원 포즈 추정에 대해서 나누어 살펴보고자 한다. 끝으로 연구에 필요한 데이터 세트 및 사람의 포즈 추정 기술을 적용하는 다양한 연구 사례를 살펴볼 것이다.

인간-컴퓨터 상호 작용을 위한 인간 팔의 3차원 자세 추정 - 기계요소 모델링 기법을 컴퓨터 비전에 적용

한영모,Han Young-Mo 대한전자공학회 2005 電子工學會論文誌-SC (System and control) Vol.42 No.4

인간은 의사 표현을 위해 음성언어 뿐 아니라 몸짓 언어(body languages)를 많이 사용한다 이 몸짓 언어 중 대표적인 것은, 물론 손과 팔의 사용이다. 따라서 인간 팔의 운동 해석은 인간과 기계의 상호 작용(human-computer interaction)에 있어 매우 중요하다고 할 수 있다. 이러한 견지에서 본 논문에서는 다음과 같은 방법으로 컴퓨터비전을 이용한 인간팔의 3차원 자세 추정 방법을 제안하다. 먼저 팔의 운동이 대부분 회전 관절(revolute-joint)에 의해 이루어진다는 점에 착안하여, 컴퓨터 비전 시스템을 활용한 회전 관절의 3차원 운동 해석 기법을 제안한다. 이를 위해 회전 관절의 기구학적 모델링 기법(kinematic modeling techniques)과 컴퓨터 비전의 경사 투영 모델(perspective projection model)을 결합한다. 다음으로, 회전 관절의 3차원 운동해석 기법을 컴퓨터 비전을 이용한 인간 팔의 3차원 자세 추정 문제에 웅용한다. 그 기본 발상은 회전 관절의 3차원 운동 복원 알고리즘을 인간 팔의 각 관절에 순서 데로 적용하는 것이다. 본 알고리즘은 특히 유비쿼터스 컴퓨팅(ubiquitous computing)과 가상현실(virtual reality)를 위한 인간-컴퓨터 상호작용(human-computer interaction)이라는 응용을 목표로, 고수준의 정확도를 갖는 폐쇄구조 형태(closed-form)의 해를 구하는데 주력한다. For expressing intention the human often use body languages as well as vocal languages. Of course the gestures using arms and hands are the representative ones among the body languages. Therefore it is very important to understand the human arm motion in human-computer interaction. In this respect we present here how to estimate 3D pose of human arms by using computer vision systems. For this we first focus on the idea that the human arm motion consists of mostly revolute joint motions, and then we present an algorithm for understanding 3D motion of a revolute joint using vision systems. Next we apply it to estimating 3D pose of human arms using vision systems. The fundamental idea for this algorithm extension is that we may apply the algorithm for a revolute joint to each of the revolute joints of hmm arms one after another. In designing the algorithms we focus on seeking closed-form solutions with high accuracy because we aim at applying them to human computer interaction for ubiquitous computing and virtual reality.

Multi-resolution Fusion Network for Human Pose Estimation in Low-resolution Images

Boeun Kim,YeonSeung Choo,Hea In Jeong,Chung-Il Kim,Saim Shin,김정호 한국인터넷정보학회 2022 KSII Transactions on Internet and Information Syst Vol.16 No.7

2D human pose estimation still faces difficulty in low-resolution images. Most existing top-down approaches scale up the target human bonding box images to the large size and insert the scaled image into the network. Due to up-sampling, artifacts occur in the low-resolution target images, and the degraded images adversely affect the accurate estimation of the joint positions. To address this issue, we propose a multi-resolution input feature fusion network for human pose estimation. Specifically, the bounding box image of the target human is rescaled to multiple input images of various sizes, and the features extracted from the multiple images are fused in the network. Moreover, we introduce a guiding channel which induces the multi-resolution input features to alternatively affect the network according to the resolution of the target image. We conduct experiments on MS COCO dataset which is a representative dataset for 2D human pose estimation, where our method achieves superior performance compared to the strong baseline HRNet and the previous state-of-the-art methods.

수치적인 역운동학 기반 UKF를 이용한 효율적인 중간 관절 추정

서융호(Yungho Seo),이준성(Junsung Lee),이칠우(Chilwoo Lee) 大韓電子工學會 2010 電子工學會論文誌-SP (Signal processing) Vol.47 No.6

영상 기반의 모션 캡처에 대한 연구는 인체의 특징 영역 검출, 정확한 자세 추정 및 실시간 성능 등의 문제를 풀기 위해 많은 연구가 진행되고 있다. 특히, 인체의 많은 관절 정보를 복원하기 위해 다양한 방법이 제안되고 있다. 본 논문에서는 수치적인 역운동학 방법의 단점을 개선한 실시간 모션 캡처 방법을 제안한다. 기존의 수치적인 역운동학 방법은 많은 반복 연산이 필요하며, 국부최소치 문제가 발생할 수 있다. 본 논문에서는 이러한 문제를 해결하기 위해 기존의 수치적인 역운동학 해법과 UKF를 결합하여 중간관절을 복원하는 방법을 제안한다. 수치적인 역운동학의 해와 UKF를 결합함으로써, 중간 관절 추정 시 최적값에 보다 안정적이고 빠른 수렴이 가능하다. 모션 캡처를 위해 먼저, 배경 차분과 피부색 검출 방법을 이용하여 인체의 특징 영역을 추출한다. 다수의 카메라로부터 추출된 2차원 인체 영역 정보로부터 3차원 정보를 복원하고, UKF와 결합된 수치적인 역운동학 해법을 통해 동작자의 중간 관절 정보를 추정한다. 수치적인 역운동학의 해는 UKF의 상태 추정 시 안정적인 방향을 제시하고, UKF는 다수의 샘플을 기반으로 최적 상태를 찾음으로써, 전역해에 보다 빠르게 수렴한다. A research of image-based articulated pose estimation has some problems such as detection of human feature, precise pose estimation, and real-time performance. In particular, various methods are currently presented for recovering many joints of human body. We propose the novel numerical inverse kinematics improved with the UKF(unscented Kalman filter) in order to estimate the human pose in real-time. An existing numerical inverse kinematics is required many iterations for solving the optimal estimation and has some problems such as the singularity of jacobian matrix and a local minima. To solve these problems, we combine the UKF as a tool for optimal state estimation with the numerical inverse kinematics. Combining the solution of the numerical inverse kinematics with the sampling based UKF provides the stability and rapid convergence to optimal estimate. In order to estimate the human pose, we extract the interesting human body using both background subtraction and skin color detection algorithm. We localize its 3D position with the camera geometry. Next, through we use the UKF based numerical inverse kinematics, we generate the intermediate joints that are not detect from the images. Proposed method complements the defect of numerical inverse kinematics such as a computational complexity and an accuracy of estimation.

Empirical Comparison of Deep Learning Networks on Backbone Method of Human Pose Estimation

( Beanbonyka Rim ),( Junseob Kim ),( Yoo-joo Choi ),( Min Hong ) 한국인터넷정보학회 2020 인터넷정보학회논문지 Vol.21 No.5

Accurate estimation of human pose relies on backbone method in which its role is to extract feature map. Up to dated, the method of backbone feature extraction is conducted by the plain convolutional neural networks named by CNN and the residual neural networks named by Resnet, both of which have various architectures and performances. The CNN family network such as VGG which is well-known as a multiple stacked hidden layers architecture of deep learning methods, is base and simple while Resnet which is a bottleneck layers architecture yields fewer parameters and outperform. They have achieved inspired results as a backbone network in human pose estimation. However, they were used then followed by different pose estimation networks named by pose parsing module. Therefore, in this paper, we present a comparison between the plain CNN family network (VGG) and bottleneck network (Resnet) as a backbone method in the same pose parsing module. We investigate their performances such as number of parameters, loss score, precision and recall. We experiment them in the bottom-up method of human pose estimation system by adapted the pose parsing module of openpose. Our experimental results show that the backbone method using VGG network outperforms the Resent network with fewer parameter, lower loss score and higher accuracy of precision and recall.

A Multi-Stage Convolution Machine with Scaling and Dilation for Human Pose Estimation

( Yali Nie ),( Jaehwan Lee ),( Sook Yoon ),( Dong Sun Park ) 한국인터넷정보학회 2019 KSII Transactions on Internet and Information Syst Vol.13 No.6

Vision-based Human Pose Estimation has been considered as one of challenging research subjects due to problems including confounding background clutter, diversity of human appearances and illumination changes in scenes. To tackle these problems, we propose to use a new multi-stage convolution machine for estimating human pose. To provide better heatmap prediction of body joints, the proposed machine repeatedly produces multiple predictions according to stages with receptive field large enough for learning the long-range spatial relationship. And stages are composed of various modules according to their strategic purposes. Pyramid stacking module and dilation module are used to handle problem of human pose at multiple scales. Their multi-scale information from different receptive fields are fused with concatenation, which can catch more contextual information from different features. And spatial and channel information of a given input are converted to gating factors by squeezing the feature maps to a single numeric value based on its importance in order to give each of the network channels different weights. Compared with other ConvNet-based architectures, we demonstrated that our proposed architecture achieved higher accuracy on experiments using standard benchmarks of LSP and MPII pose datasets.