Online Video Synopsis via Multiple Object Detection

JaeWon Lee(이재원),DoHyeon Kim(김도현),Yoon Kim(김윤) 한국컴퓨터정보학회 2019 韓國컴퓨터情報學會論文誌 Vol.24 No.8

In this paper, an online video summarization algorithm based on multiple object detection is proposed. As crime has been on the rise due to the recent rapid urbanization, the people’s appetite for safety has been growing and the installation of surveillance cameras such as a closed-circuit television(CCTV) has been increasing in many cities. However, it takes a lot of time and labor to retrieve and analyze a huge amount of video data from numerous CCTVs. As a result, there is an increasing demand for intelligent video recognition systems that can automatically detect and summarize various events occurring on CCTVs. Video summarization is a method of generating synopsis video of a long time original video so that users can watch it in a short time. The proposed video summarization method can be divided into two stages. The object extraction step detects a specific object in the video and extracts a specific object desired by the user. The video summary step creates a final synopsis video based on the objects extracted in the previous object extraction step. While the existed methods do not consider the interaction between objects from the original video when generating the synopsis video, in the proposed method, new object clustering algorithm can effectively maintain interaction between objects in original video in synopsis video. This paper also proposed an online optimization method that can efficiently summarize the large number of objects appearing in long-time videos. Finally, Experimental results show that the performance of the proposed method is superior to that of the existing video synopsis algorithm.

중요도 기반 퍼지 원 클래스 서포트 벡터 머신을 이용한 비디오 요약 기술

김기주 ( Kijoo Kim ),최영식 ( Youngsik Choi ) 한국인터넷정보학회 2011 인터넷정보학회논문지 Vol.12 No.5

본 논문에서는 비디오 요약을 시각적으로 특징이 있고 주관적으로 중요한 비디오 세그먼트 집합을 구하는 새로운 요약방식을 기술한다. 시각적으로 특징이 있는 데이터 포인트를 찾기 위해 novelty detection으로 잘 알려져 있는 OC-SVM(One-Class Support Vector Machine)을 사용할 수 있다. 그러나 OC-SVM의 처리과정에 비디오 세그먼트에 대한 사용자의 주관적인 중요도를 반영하기는 어렵다. OC-SVM의 처리과정에 사용자의 주관적 중요성을 반영하기 위해서, 본 논문에서는 OC-SVM의 퍼지 버전을 유도한다. IFOC-SVM(Importance-based Fuzzy One-Class Support Vector Machine)은 비디오 세그먼트의 중요도에 따라 각 데이터 포인트에 가중치를 부여하고 데이터 분포의 서포트를 측정한다. 이때, 구해진 서포트 벡터는 비디오 세그먼트의 중요도와 시각적 특징 관점에서 비디오의 내용을 축약하여 표현한다. 제안된 알고리즘의 성능을 증명하기 위하여 가상의 데이터들과 다양한 종류의 비디오들을 가지고 실험하였다. 실험 결과는 제안하는 방법의 성능이 다른 비디오요약의 성능보다 우수함을 보여주었다. In this paper, we address a video summarization task as generating both visually salient and semantically important video segments. In order to find salient data points, one can use the OC-SVM (One-class Support Vector Machine), which is well known for novelty detection problems. It is, however, hard to incorporate into the OC-SVM process the importance measure of data points, which is crucial for video summarization. In order to integrate the importance of each point in the OC-SVM process, we propose a fuzzy version of OC-SVM. The Importance-based Fuzzy OC-SVM weights data points according to the importance measure of the video segments and then estimates the support of a distribution of the weighted feature vectors. The estimated support vectors form the descriptive segments that best delineate the underlying video content in terms of the importance and salience of video segments. We demonstrate the performance of our algorithm on several synthesized data sets and different types of videos in order to show the efficacy of the proposed algorithm. Experimental results showed that our approach outperformed the well known traditional method.

내용기반 비디오 요약을 위한 효율적인 얼굴 객체 검출

김종성,이순탁,백중환,Kim Jong-Sung,Lee Sun-Ta,Baek Joong-Hwan 한국통신학회 2005 韓國通信學會論文誌 Vol.30 No.7C

본 논문에서는 효율적인 얼굴 영역 검출 기법을 제안하고 얼굴 객체 검출을 통해 인물 기반의 비디오 시스템을 제공한다. 비디오 분할을 위해 비디오 시퀀스로부터 장면 전환점을 검출하고 분할된 장면들로부터 대표 프레임을 선정한다. 대표 프레임은 인접 프레임 간 변화량이 가장 적은 프레임으로 선정하였으며 추출된 대표 프레임에 대해서 얼굴 영역 검출 알고리즘을 적용하여 등장인물을 포함하는 프레임들을 정보로 제공한다. 얼굴영역 검출을 위해 피부색의 통계적 특성을 이용한 Bayes 분류기를 이용한다. 피부색 검출 결과 영상으로부터 수직 및 수평 투영 기법을 이용하여 영상 분할을 수행하고 후보군들을 생성한다. 생성된 후보군 중 오검출 영역을 최소화하기 위해서 이진 분류 나무(CART)를 이용하여 분류기를 생성한다. 특징 값으로는 SGLD(spatial gray level dependence) 매트릭스로부터 Inertial, Inverse Difference, Correlation 등의 질감 정보를 이용하여 최적의 이진 분류 나무를 생성한다. 실험 결과 제안된 얼굴 영역 검출 알고리즘은 복잡하고 다양한 배경에서도 우수한 성능을 보였으며, 얼굴 객체를 포함하는 프레임들을 비디오 정보로 제공한다. 제안하는 시스템은 향후 화자 인식 기법을 이용하여 등장인물 기반의 비디오 분석 및 에 활용될 수 있을 것이다. In this paper, we propose an efficient face region detection technique for the content-based video summarization. To segment video, shot changes are detected from a video sequence and key frames are selected from the shots. We select one frame that has the least difference between neighboring frames in each shot. The proposed face detection algorithm detects face region from selected key frames. And then, we provide user with summarized frames included face region that has an important meaning in dramas or movies. Using Bayes classification rule and statistical characteristic of the skin pixels, face regions are detected in the frames. After skin detection, we adopt the projection method to segment an image(frame) into face region and non-face region. The segmented regions are candidates of the face object and they include many false detected regions. So, we design a classifier to minimize false lesion using CART. From SGLD matrices, we extract the textual feature values such as Inertial, Inverse Difference, and Correlation. As a result of our experiment, proposed face detection algorithm shows a good performance for the key frames with a complex and variant background. And our system provides key frames included the face region for user as video summarized information.

Video Summarization using Low-Rank Sparse Representation

Hyuncheol Kim,Joonki Paik 대한전자공학회 2018 IEIE Transactions on Smart Processing & Computing Vol.7 No.3

Extraction of key frames plays a significant role in selecting the most informative subset of frames from a huge amount of video data in order to compress its content. Key frame extraction has a good number of applications, such as video browsing, indexing, and storage. Most key framebased video summarization techniques directly process an input video dataset. An alternative approach uses only low-rank components, without considering the rest of the significant information in the video. This paper presents a novel key frame-extraction framework based on low-rank sparse representation. The proposed framework is motivated by the fact that low-rank sparse-feature representation pursues consistent nonlocal structures for image pixels with similar features. We use low-rank representation to ensure globally consistent non-salient systematic structures for pixels with similar features, and we also use sparse representation to robustly select the best sample for distinct structures of all pixels. Experimental results on a human-labeled benchmark dataset and a comparative performance evaluation with state-of-the-art methods demonstrate the advantages of the proposed method.

실시간 뇌파반응을 이용한 주제관련 영상물 쇼트 자동추출기법 개발연구

김용호(Yong Ho Kim),김현희(Hyun Hee Kim) 한국멀티미디어학회 2016 멀티미디어학회논문지 Vol.19 No.8

To obtain good summarization algorithms, we need first understand how people summarize videos. Semantic gap refers to the gap between semantics implied in video summarization algorithms and what people actually infer from watching videos. We hypothesized that ERP responses to real time videos will show either N400 effects to topic-irrelevant shots in the 300∼500ms time-range after stimulus on-set or P600 effects to topic-relevant shots in the 500∼700ms time range. We recruited 32 participants in the EEG experiment, asking them to focus on the topic of short videos and to memorize relevant shots to the topic of the video. After analysing real time videos based on the participants’ rating information, we obtained the following t-test result, showing N400 effects on PF1, F7, F3, C3, Cz, T7, and FT7 positions on the left and central hemisphere, and P600 effects on PF1, C3, Cz, and FCz on the left and central hemisphere and C4, FC4, P8, and TP8 on the right. A further 3-way MANOVA test with repeated measures of topic-relevance, hemisphere, and electrode positions showed significant interaction effects, implying that the left hemisphere at central, frontal, and pre-frontal positions were sensitive in detecting topic-relevant shots while watching real time videos.

실시간 뇌파반응을 이용한 주제관련 영상물 쇼트 자동추출기법 개발연구 - ERP의 N400 및 P600 가설검증을 중심으로

김용호,김현희 한국멀티미디어학회 2016 멀티미디어학회논문지 Vol.19 No.8

To obtain good summarization algorithms, we need first understand how people summarize videos. 'Semantic gap' refers to the gap between semantics implied in video summarization algorithms and what people actually infer from watching videos. We hypothesized that ERP responses to real time videos will show either N400 effects to topic-irrelevant shots in the 300∼500ms time-range after stimulus on-set or P600 effects to topic-relevant shots in the 500∼700ms time range. We recruited 32 participants in the EEG experiment, asking them to focus on the topic of short videos and to memorize relevant shots to the topic of the video. After analysing real time videos based on the participants’ rating information, we obtained the following t-test result, showing N400 effects on PF1, F7, F3, C3, Cz, T7, and FT7 positions on the left and central hemisphere, and P600 effects on PF1, C3, Cz, and FCz on the left and central hemisphere and C4, FC4, P8, and TP8 on the right. A further 3-way MANOVA test with repeated measures of topic-relevance, hemisphere, and electrode positions showed significant interaction effects, implying that the left hemisphere at central, frontal, and pre-frontal positions were sensitive in detecting topic-relevant shots while watching real time videos.

스코어 정보를 이용한 농구 비디오의 자동요약

정철곤,김의진,이광국,김회율,Jung, Cheol-Kon,Kim, Eui-Jin,Lee, Gwang-Gook,Kim, Whoi-Yul 한국통신학회 2007 韓國通信學會論文誌 Vol.32 No.9c

본 논문에서는 농구 비디오의 내용기반 자동 요약 방법을 제안하였다. 의미 있는 요약을 위해, 농구 비디오에 포함되어 있는 스코어 정보를 이용하였다. 스코어 정보는 비디오에 포함된 스코어 자막의 숫자들을 인식한 후, 인식된 스코어의 변화를 분석함에 의해 획득된다. 일반적으로 농구경기의 중요한 이벤트는 3점 슛, 일방적 리드, 역전 등이다. 제안된 요약방법은 스코어 정보를 활용해 이러한 이벤트들을 검출하였으며, 이를 기반으로 농구 비디오의 요약 및 하이라이트를 생성하였다. In this paper, we proposed a method for content based automatic summarization of basketball game videos. For meaningful summary, we used the score information in basketball videos. And the score information is obtained by recognizing the digits on the score caption and analyzing the variation of the score. Generally, important events of basketball are the 3-point shot, one-sided runs, the lead changes, and so on. We have detected these events using score information and made summaries and highlights of basketball video games.

스코어 정보를 이용한 농구 비디오의 자동요약

정철곤,김의진,이광국,김회율,Jung, Cheol-Kon,Kim, Eui-Jin,Lee, Gwang-Gook,Kim, Whoi-Yul 한국통신학회 2007 韓國通信學會論文誌 Vol.32 No.8c

본 논문에서는 농구 비디오의 내용기반 자동 요약 방법을 제안하였다. 의미 있는 요약을 위해, 농구 비디오에 포함되어 있는 스코어 정보를 이용하였다. 스코어 정보는 비디오에 포함된 스코어 자막의 숫자들을 인식한 후, 인식된 스코어의 변화를 분석함에 의해 획득된다. 일반적으로 농구경기의 중요한 이벤트는 3점 슛, 일방적 리드, 역전 등이다. 제안된 방법은 스코어 정보를 활용해 이러한 이벤트들을 검출하였으며, 이를 기반으로 농구 비디오의 요약 및 하이라이트를 생성하였다. In this paper, we proposed a method for content based automatic summarization of basketball game videos. For meaningful summary, we used the score information in basketball videos. And the score information is obtained by recognizing the digits on the score caption and analyzing the variation of the score. Generally, important events of basketball are the 3-point shot, one-sided runs, the lead changes, and so on. We have detected these events using score information and made summaries and highlights of basketball video games.

Improved Quality Keyframe Selection Method for HD Video

( Hyeon Seok Yang ),( Jong Min Lee ),( Woojin Jeong ),( Seung-hee Kim ),( Sun-joong Kim ),( Young Shik Moon ) 한국인터넷정보학회 2019 KSII Transactions on Internet and Information Syst Vol.13 No.6

With the widespread use of the Internet, services for providing large-capacity multimedia data such as video-on-demand (VOD) services and video uploading sites have greatly increased. VOD service providers want to be able to provide users with high-quality keyframes of high quality videos within a few minutes after the broadcast ends. However, existing keyframe extraction tends to select keyframes whose quality as a keyframe is insufficiently considered, and it takes a long computation time because it does not consider an HD class image. In this paper, we propose a keyframe selection method that flexibly applies multiple keyframe quality metrics and improves the computation time. The main procedure is as follows. After shot boundary detection is performed, the first frames are extracted as initial keyframes. The user sets evaluation metrics and priorities by considering the genre and attributes of the video. According to the evaluation metrics and the priority, the low-quality keyframe is selected as a replacement target. The replacement target keyframe is replaced with a high-quality frame in the shot. The proposed method was subjectively evaluated by 23 votes. Approximately 45% of the replaced keyframes were improved and about 18% of the replaced keyframes were adversely affected. Also, it took about 10 minutes to complete the summary of one hour video, which resulted in a reduction of more than 44.5% of the execution time.

중간값 필터와 신경망 회로를 사용한 자동 컷 검출 알고리즘

전승철,박성한,Jun, Seung-Chul,Park, Sung-Han 대한전자공학회 2002 電子工學會論文誌-SP (Signal processing) Vol.39 No.4

본 논문은 MPEG 스트림 데이터에서 효과적으로 화면 전환 경계를 찾아내는 알고리즘을 제안한다. 이를 위하여 먼저 연속적인 장면의 변화 정도를 표시하는 척도로써 히스토그램 차이 값(histogram difference value)과 픽셀 차이 값(pixel difference value)을 각각 하나의 신호로 취급한다. 이 신호에 중간 값 필터를 적용하여 얻어진 값과 원래의 신호의 차이값인 MFD(Median filtered difference) 값을 구한다. 이렇게 얻어진 MFD의 값이 크면 회면 전환이 일어남을 나타내며 따라서 컷 검출의 기준이 될 수 있다. 또한, 인공 신경망을 사용하여 컷 경계가 되는 MFD값의 문턱치를 결정한다. 제안된 알고리즘은 변화량이 심한 동영상이나 급작스럽게 밝아지는 프레임을 포함하는 동영상에서 적절히 컷 전환을 검출함을 보여 준다. 실험결과에서 제안된 알고리즘의 성능을 보여준다. In this paper, an efficient method to find shot boundaries in the MPEG video stream data is proposed. For this purpose, we first assume that the histogram difference value(HDV) and pixel difference value(PDV) as an one dimensional signal and apply the median filter to these signals. The output of the median filter is subtracted from the original signal to produce the median filtered difference(MFD). The MFD is a criterion of shot boundary. In addition a neural network is employed and trained to find exactly cut boundary. The proposed algorithm shows that the cut boundaries are well extracted, especially in a dynamic video.