미디어 인공지능 : 컴퓨터 비전 관련 딥러닝 모델의 미디어 동영상 분야 적용 가능성에 관한 연구

박대민 한국언론학회 2022 커뮤니케이션 이론 Vol.18 No.1

Recently, media researchers employ deep learning models related to computer vision to perform automated content analysis of videos. Understanding deep learning models is also essential to AI(artificial intelligence) driven digital transformation in the media industry. In this paper, we reviewed computer vision-related deep learning models that are widely used for video analytics and generation. First, we looked at convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are widely used for classification, as well as generative adversarial network (GANs) and autoencoders (AEs) as generation models, and transfer learning using pre-training models. Following that, we proposed tasks in five major categories for which AI could be highly useful: object detection, action recognition, event detection, video summarization, and video classification. Then, for subtasks such as object detection, face recognition, facial expression recognition, landmark recognition, product recognition, pose estimation, anomaly detection, context recognition, video summarization, and video classification, we introduced state-of-the-arts (SOTAs) and benchmark datasets. Finally, the potential academic and practical applications of these models were discussed. We anticipate that media researchers or media service providers will understand the major trend of deep learning in computer vision and will be able to get knowledge when using deep learning models independently or collaborating with programmers. 미디어 동영상 분야는 컴퓨터 비전 관련 딥러닝 모델을 활용해 연구 차원에서는 동영상의 자동화된 내용분석을 수행하고 실무 차원에서는 미디어 분야의 디지털 전환을 통해 서비스를 개선할 여지가 큰 영역이다. 이 논문에서는 미디어 동영상의 분석과 생성에 활용도가 높은 비전 관련 딥러닝 기반 모델을 검토했다. 우선 다양한 모델의 기축이 되는 알고리즘으로서 분류 모델로 널리 사용되는 합성곱 신경망(CNN)과 순환 신경망(RNN), 생성 모델로 사용되는 적대적 생성 신경망(GAN)과 오토인코더(AE), 사전 훈련 모델을 활용하는 전이학습을 살펴보았다. 다음으로 미디어 동영상 영역에서 활용도가 높은 과업을 객체탐지, 행동인식, 사건탐지, 동영상 요약, 동영상 분류 등 5개 대분류와 객체탐지, 안면인식, 표정인식, 랜드마크 인식, 상품인식, 행동인식, 자세추정, 이상탐지, 상황인식, 동영상 요약, 동영상 분류 등 11개 소분류로 제시했다. 이어 각 과업별 SOTA(state-of-the-art)와 벤치마크 데이터셋을 소개했다. 끝으로 이러한 모델의 학문적, 실무적 활용 가능성을 제시해보았다. 본 논문은 수식이나 프로그래밍에 대한 지식이 없이 미디어 연구자나 미디어 서비스 기획자가 비전 분야 딥러닝의 큰 흐름을 파악하고 관련 모델을 직접 활용하거나 컴퓨터공학 분야의 연구자 또는 개발자와 협업할 때 배경지식을 제공할 것으로 기대한다. 또한 비전 관련 딥러닝이 발전함에 따라 미디어 인공지능 기반 동영상 빅데이터 분석 시스템의 개발 가능성도 높아질 것이다.

Predicting Learning Achievements with Indicators of Perceived Affordances Based on Different Levels of Content Complexity in Video-based Learning

Dasom KIM,Gyeoun JEONG The Korean Society for Education Technology 2024 Educational Technology International Vol.25 No.1

The purpose of this study was to identify differences in learning patterns according to content complexity in video-based learning environments and to derive variables that have an important effect on learning achievement within particular learning contexts. To achieve our aims, we observed and collected data on learners' cognitive processes through perceived affordances, using behavioral logs and eye movements as specific indicators. These two types of reaction data were collected from 67 male and female university students who watched two learning videos classified according to their task complexity through the video learning player. The results showed that when the content complexity level was low, learners tended to navigate using other learners' digital logs, but when it was high, students tended to control the learning process and directly generate their own logs. In addition, using derived prediction models according to the degree of content complexity level, we identified the important variables influencing learning achievement in the low content complexity group as those related to video playback and annotation. In comparison, in the high content complexity group, the important variables were related to active navigation of the learning video. This study tried not only to apply the novel variables in the field of educational technology, but also attempt to provide qualitative observations on the learning process based on a quantitative approach.

For the Right Time: Monitoring Weekly Changes of Prediction Power in Video-based Learning Environment

( Yeonjeong Park ),( Jihyun Yu ),( Il Hyun Jo ) 한국교육공학회 2016 한국교육공학회 학술대회발표자료집 Vol.2016 No.1

Time is a very important and useful measurement unit in the field of learning analytics. It implies not only physical time spent on a specific learning activity but also time measuring how frequently specific learning-related actions had occurred, it can be further extended to conceptualize how regularly one``s behaviors occurred. Moreover, it is a pivotal consideration when providing students with feedback as a type of learning facilitation. By reviewing these diverse perspectives encompassing "time", this study focused on the prediction of time throughout observable time-related variables. In other words, this study aimed to find the best time to predict students`` learning performance over a period of 15 weeks. For this purpose, we monitored 380 online students`` weekly learning behavioral patterns with their time stamps remaining in the video-based learning environment. There were a total of 16 accumulated weekly variables including three kinds of time-related concepts, duration, frequency, and regularity inserted to generate 15 weekly prediction models for one semester. The changes of prediction power measured by an adjusted R2 score and significant independent variables to contribute the model were observed in two cases dealing with the same subject matter. Findings allow audiences to discuss what might be the right time and feedback to inform students the results of learning analytics.

실시간 영상인식 국방 빅데이터 플랫폼(D-NET) 개발에 대한 연구

장동인 한국방위산업학회 2016 韓國防衛産業學會誌 Vol.23 No.1

영상 빅데이터를 인덱싱하고 검색하고 분석하는 시도는 DARPA, Microsoft, IBM, 버지니아 대학등 많이 있었으나, 실시간으로 영상 빅데이터를 처리하여 분석하고, 검색하며 GPS정보를 활용하여 영상에 나타난 목표 객체를 GIS에서 추적하는 종합적인 실시간 분석 시스템은 아직은 없다. 실시간 영상인식 국방빅데이터 플랫폼(D-NET) 개발은 현재 국방과학연구소에서 수행중인 과제로서 전반적인 설계가 완료가 된 상태이다. 이 연구논문은 D-NET에 대한 아키텍처적인 다양한 결정요소와 실제 기능에 대한 소개를 하는 것이 목적이다. There were many attempts to index, search and analyze image big data from DARPA, Microsoft, IBM, University of Virginia and other institutes, but these attempts to build a system to process, analyze, search, and track a target object in real-time appeared in thousands of screens have not produced promising results. This paper introduces current efforts to build “Defense Big data Platform for Real-time video analytics” (D-NET) and points out the architectural designs and various functionalities.

AN AUTOMATIC VIDEO ANALYTICS APPROACH TO EXAMINE THE EFFECTS OF VIDEO CONTENT FEATURES ON AUDIENCE ENGAGEMENT

Jingjie Zhu,Mingming Cheng,Ying Wang,Jie Tan 글로벌지식마케팅경영학회 2023 Global Marketing Conference Vol.2023 No.07

While there is an increasing number of studies highlighting the power of videos in influencing audience attitudes and behavior, academic research in tourism is largely behind due to the methodological challenges of analyzing unstructured video data. This study adopts an automatic video analytics approach to examine the relationship between content features of pro-environmental videos and audience engagement in tourism. Artificial intelligence was used to extract video content features by detecting scenes and shots as well as labels (e.g., trees). Our findings suggest that there exists an inverted U-shape relationship between video informativeness and audience engagement. This study makes significant theoretical and methodological contributions to extant tourism literature by theoretically explaining and empirically testing how video content features influence audience engagement in pro-environmental video communications in tourism.

동영상 강의에서 메타인지 지원이 학습결과 예측에 미치는 영향: 다중양식 학습분석을 중심으로

함윤희 ( Yoonhee Ham ),조영환 ( Young Hoan Cho ),김혁기 ( Hyug Gi Kim ),이재용 ( Jaeyong Lee ),김혜은 ( Hyeeun Kim ),이한솔 ( Hansol Lee ) 한국교육정보미디어학회 2021 교육정보미디어연구 Vol.27 No.4

코로나19 팬데믹으로 인해 비대면 교육이 증가하면서 동영상 강의의 중요도가 높아졌다. 동영상 강의의 효과를 높이기 위해서는 교수자가 학습결과를 사전에 예측하여 학습 지원을 적시에 제공하는 것이 필요하다. 최근에는 학습결과 예측의 정확도를 높이기 위해서 생리심리데이터를 포함하여 다양한 학습 데이터를 통합적으로 수집하고 분석하는 다중양식 학습분석 연구가 증가하고 있다. 하지만 예측 모델이 다양한 학습 맥락에 일반화 가능한지에 대한 연구는 부족하다. 본 연구는 동영상 학습결과를 예측하는 주요 요인인 메타인지 활동에 초점을 두고, 동영상 강의에서 메타인지 지원 여부가 학습결과 예측 모델에 어떠한 영향을 미치는지 탐구하고자 한다. 이를 위해 총 33명의 학습자를 메타인지 지원 집단(n=17)과 메타인지 미지원 집단(n=16)으로 무선 할당하고, 개별적으로 약 40분 동안 인체의 삼투압에 대한 동영상 강의를 학습하도록 하였다. 다중양식 학습분석을 위해 동영상 학습 과정을 영상으로 촬영하고, 손목밴드를 이용하여 피부전도도를 측정하고, 설문지로 인지부하를 측정하고, 사전 및 사후 검사를 실시하였다. 이렇게 수집된 데이터와 다양한 머신러닝 알고리즘을 이용해서 학습결과 상·하 집단을 예측하는 모델을 만들고, 각 모델의 성능을 평가하여 최적의 예측 모델을 도출하였다. 연구 결과, 메타인지 지원여부에 따라 최적의 동영상 학습결과 예측 모델이 서로 상이하게 나타났다. 메타인지 미지원 집단에서는 메타인지 활동과 피부전도도로 측정한 각성도가 최적 모델의 예측 변인으로 나타났지만, 메타인지 지원 집단에서는 인지부하와 사전지식 수준이 최적 모델의 예측 변인으로 나타났다. 본 연구는 메타인지 지원과 같은 교수 방법의 변화가 예측 모델에 주요한 영향을 미칠 수 있음을 보여주며, 이는 예측 모델을 새로운 맥락에 적용할 때 많은 주의가 필요하다는 것을 시사한다. Video lectures have played an important role during the COVID-19 pandemic, which significantly increased remote learning in school. To enhance the effectiveness of video lectures, instructors should predict students’ learning outcomes and provide learning supports at the right time. Recently, multimodal learning analytics has been developed in order to accurately predict learning outcomes using diverse learning data like psychophysiological measures. Nevertheless, there is a lack of research on whether a prediction model is generalizable across diverse learning contexts using different instructional methods. This research intends to explore the influence of metacognitive support on the prediction model of learning outcomes in video lectures. A total of 33 learners were randomly assigned to metacognitive support (n=17) and non-support (n=16) groups, and they individually studied video lectures of the osmotic pressure in the human body for 40 minutes. This study recorded the learning process with video, measured skin conductance responses with a wristband, collected the data of cognitive load using a survey, and conducted pre- and post-tests. Using the data collected, machine learning algorithms generated models to predict high- and low-achievement groups, and an optimal prediction model was selected through evaluating the performance of the models. This study found that the optimal prediction model of the metacognitive support group was different from that of the non-support group. In the non-support group, metacognitive activities and arousal levels measured by skin conductance responses were included in the optimal prediction model, whereas cognitive load and prior knowledge were included in the optimal prediction model of the metacognitive support group. This study shows that instructional methods like metacognitive support can significantly influence predicting learning outcomes in video lectures, so prediction models should be carefully applied to novel learning contexts.

Scalable Big Data Pipeline for Video Stream Analytics Over Commodity Hardware

Umer Ayub,Syed M. Ahsan,Shavez M. Qureshi 한국인터넷정보학회 2022 KSII Transactions on Internet and Information Syst Vol.16 No.4

A huge amount of data in the form of videos and images is being produced owning to advancements in sensor technology. Use of low performance commodity hardware coupled with resource heavy image processing and analyzing approaches to infer and extract actionable insights from this data poses a bottleneck for timely decision making. Current approach of GPU assisted and cloud-based architecture video analysis techniques give significant performance gain, but its usage is constrained by financial considerations and extremely complex architecture level details. In this paper we propose a data pipeline system that uses open-source tools such as Apache Spark, Kafka and OpenCV running over commodity hardware for video stream processing and image processing in a distributed environment. Experimental results show that our proposed approach eliminates the need of GPU based hardware and cloud computing infrastructure to achieve efficient video steam processing for face detection with increased throughput, scalability and better performance.

Analysis of Urban Transit Environment for Video Analytics Application and Processing Scenarios

Kwang-Young Park,Goo-Man Park 보안공학연구지원센터 2012 International Journal of Security and Its Applicat Vol.6 No.2

드론 영상 종합정보처리 및 분석용 시스템 개발

임송원,조성만,박구만 한국방송∙미디어공학회 2019 방송공학회논문지 Vol.24 No.2

본 논문에서는 다양한 재난치안안전 임무 상황에서 적용할 수 있는 드론 영상 종합정보 처리 및 분석용 시스템을 제안한다. 제안하는 시스템은 드론에서 획득한 영상을 서버에 저장하고, 다양한 시나리오에 따른 영상 처리 및 분석을 수행한다. 각 임무에 따라 필요한 기능은 딥러닝을 활용하여 드론으로부터 확보하는 영상에서 영상분석 시스템을 구성한다. 실험 영상을 통해 교통량 측정, 용의자 및 차량 추적, 조난자 식별 및 해상 초계 임무에 적용할 수 있음을 확인했다. 드론 운용자가 임무에 따른 필요 기능을 선택하고 신속하게 대처할 수 있는 시스템을 구현하였다.

CCTV 카메라의 인구 이동량 측정을 이용한 빌딩 에너지 절약시스템

류홍남(Hong-Nam Ryu),류경모(Gyeong-Mo Yoo),최병욱(Byoung-Wook Choi) 한국조명·전기설비학회 2016 조명·전기설비학회논문지 Vol.30 No.8

In general, IP (Internet Protocol) Cameras are CCTVs that analyzes image data not only for security purposes, but also for measuring presence and counting of people with video analytic functions. In this paper, we present a feasibility study in using the obtained video analytic data for a building energy management system. First, we developed a simulator based on a building model where CCTV may be placed to observe people. The data from each CCTV are analyzed to produce people counting at each area of the building model. In order to analyze the energy trend of an actual building, we performed a simulation using an energy analysis program, eQUEST. The produced people counting were used to control building management system defined in eQUEST. The feasibility of a building energy management system using the analyzed image data was proved by simulation.