Big data platform for health monitoring systems of multiple bridges

Wang, Manya,Ding, Youliang,Wan, Chunfeng,Zhao, Hanwei Techno-Press 2020 Structural monitoring and maintenance Vol.7 No.4

At present, many machine leaning and data mining methods are used for analyzing and predicting structural response characteristics. However, the platform that combines big data analysis methods with online and offline analysis modules has not been used in actual projects. This work is dedicated to developing a multifunctional Hadoop-Spark big data platform for bridges to monitor and evaluate the serviceability based on structural health monitoring system. It realizes rapid processing, analysis and storage of collected health monitoring data. The platform contains offline computing and online analysis modules, using Hadoop-Spark environment. Hadoop provides the overall framework and storage subsystem for big data platform, while Spark is used for online computing. Finally, the big data Hadoop-Spark platform computational performance is verified through several actual analysis tasks. Experiments show the Hadoop-Spark big data platform has good fault tolerance, scalability and online analysis performance. It can meet the daily analysis requirements of 5s/time for one bridge and 40s/time for 100 bridges.

사물인터넷 환경을 위한 하둡 기반 빅데이터 처리 플랫폼 설계 및 구현

허석렬,이호영,이완직 한국멀티미디어학회 2019 멀티미디어학회논문지 Vol.22 No.2

In the information society represented by the Fourth Industrial Revolution, various types of data and information that are difficult to see are produced, processed, and processed and circulated to enhance the value of existing goods. The IoT(Internet of Things) paradigm will change the appearance of individual life, industry, disaster, safety and public service fields. In order to implement the IoT paradigm, several elements of technology are required. It is necessary that these various elements are efficiently connected to constitute one system as a whole. It is also necessary to collect, provide, transmit, store and analyze IoT data for implementation of IoT platform. We designed and implemented a big data processing IoT platform for IoT service implementation. Proposed platform system is consist of IoT sensing/control device, IoT message protocol, unstructured data server and big data analysis components. For platform testing, fixed IoT devices were implemented as solar power generation modules and mobile IoT devices as modules for table tennis stroke data measurement. The transmission part uses the HTTP and the CoAP, which are based on the Internet. The data server is composed of Hadoop and the big data is analyzed using R. Through the emprical test using fixed and mobile IoT devices we confirmed that proposed IoT platform system normally process and operate big data.

학술정보 영역의 빅데이터 플랫폼 이해 및 구축 전략

이원상 ( Won Sang Lee ) 한국디지틀도서관포럼 2014 디지틀 도서관 Vol.74 No.-

최근, 학술정보 영역에서 빅데이터 활용의 필요성이 제기되고 있는 가운데, 대학도서관에는 학술정보 영역의 다양한 유형의 데이터에 대한 효과적인 관리와 유통의 역할이 기대되고 있다. 이에 따라 대학도서관에서 학술정보 데이터에 대한 전략을 수립하고 체계적인 관리를 이끌 수 있는 큐레이션을 통해 보다 좋은 정보를 선별하고 제공하는 필요성이 증가하고 있다. 효과적인 데이터 큐레이션에 필수적인 요소 중 하나는, 학술정보 영역의 빅데이터를 처리할 수 있는 플랫폼에 대한 이해와 구축이다. 이 논문에서는 학술정보 빅데이터플랫폼 구축을 위한 기술을 이해하고, 오픈소스 Hadoop과 R을 기반으로 한 데이터플랫폼 구성을 제시하며, 플랫폼이 학술정보 영역의 데이터의 처리에 활용되는 사례를 살펴보았다. 이러한 시도가 향후 학술정보 빅데이터플랫폼에 대한 지속적인 논의와 시도에기여할 것으로 기대된다. Recently, university libraries are expected to play the important role on effectively managing and utilizing various data in academia. It can bring the necessity of big data curation to university libraries. Since it enables the establishment of data strategy and systematic management, data curation can provide the qualified information to researchers. Because the effective data curation can be based on the understanding and constructing the data platform, this paper reviews the technological aspect on big data platform for library and proposes the platform architecture based on Hadoop and R. The case of applying data platform is, then, provided for processing the data in academia. It is expected that the findings of this paper can contribute to the further discussion and efforts on developing the big data platform in university library.

Research on Parallel Algorithm Based On Hadoop Distributed Computing Platform

Guo Weiwei,Liu Feng 보안공학연구지원센터 2015 International Journal of Grid and Distributed Comp Vol.8 No.4

With the rapid development of the 3G network, traditional calculation methods are unable to adapt to the data scene that telecom users' Network access behavior's data scale increase rapidly dozens of TB. The cloud techniques such as Hadoop platform are introduced to solve the data storage problem. The appropriate data mining algorithms are designed from the perspective of practical application. This paper improves the traditional decision tree SPRINT algorithms, proposes a parallel computing program and successfully applies to the Hadoop platform.

CPU-GPU 이기종 플랫폼에서 하둡 맵리듀스의 가속

이새한슬(Sae-han-seul Yi),이영민(Youngmin Yi) 한국정보과학회 2014 정보과학회 컴퓨팅의 실제 논문지 Vol.20 No.6

빅데이터 시대가 도래함에 따라 하둡 맵리듀스와 같은 응용이 널리 사용되고 있다. 한편, 최근 GPGPU가 보편화되면서, 다양한 분야의 응용들이 GPU를 이용하여 가속되고 있다. 본 논문은 하둡 맵리 듀스에서 GPU를 사용하는 방법을 제안하고, CPU와 GPU가 모두 포함되는 이기종 서버로 구성된 분산환경에서 최적의 데이터 처리 속도를 얻기 위해, CPU와 GPU에 각각 할당되는 맵 태스크들에 대한 정적분할 및 동적 스케줄링에 대한 기법을 제안하였다. 노드마다 12개의 CPU 코어와 1개의 GPU가 장착된 14-노드 클러스터 환경에서 하둡 맵리듀스로 CKY 파서 응용을 수행하여, CPU 코어 1개만 사용한 단일 서버에서의 수행시간 대비 245배 가속을 하였고, 노드별로 GPU를 사용하지 않고 CPU 코어 12개만 활용하는 동일 하둡 클러스터에서의 수행시간 대비 2.5배 가속을 하였다. 또한 제안하는 기법으로 CPU 코어 12개와 GPU를 모두 사용하는 하둡 클러스터 수행시간 대비 총 2.8배 가속이 되었다. These days, big data computing is prevalent and Hadoop MapReduce framework is widely used for its simple programming model. On the other hand, General-Purpose Graphics Processing Unit (GPGPU) has become very popular and various domains of applications have been successfully accelerated using GPUs. In this paper, we propose a method to use GPU within Hadoop MapReduce framework. Then, we propose a static partitioning method that considers different capability of CPU mappers and GPU mappers, and a dynamic scheduling method that deals with a dynamic input size. Compared to a single CPU execution time, the CKY parser on a 14-node Hadoop cluster with 12 CPU cores and 1 GPU per node achieves 245 times speedup. Compared to the execution time on a 14-node Hadoop cluster with 12 CPU cores and no GPU per node, it also achieves 2.5 times speedup. Our proposed approach for both CPU and GPU mapper execution leads to an additional speedup, resulting in total of 2.8 times speedup.

하둡 및 스파크 기반 빅데이터 플랫폼을 이용한 선박 운항 효율 이상 상태 분석

이태현(Taehyeon Lee),유은섭(Eun-seop Yu),박개명(Kaemyoung Park),유성상(Seongsang Yu),박진표(Jinpyo Park),문두환(Duhwan Mun) 한국기계가공학회 2019 한국기계가공학회지 Vol.18 No.6

To reduce emissions of marine pollutants, regulations are being tightened around the world. In the shipbuilding and shipping industries, various countermeasures are being put forward. As there are limits to applying countermeasures to ships already in operation, however, it is necessary for these vessels to use energy efficiently. The sensors installed on ships typically gather a very large amount of data, and thus a big data platform is needed to manage and analyze the data. In this paper, we build a big data analysis platform based on Hadoop and Spark, and we present a method to detect abnormal ship operation using the platform. We also utilize real ship operation data to discuss the data analysis experiment.

Design of a Large-scale Task Dispatching & Processing System based on Hadoop

김직수(Jik-Soo Kim),구엔 카오(Nguyen Cao),김서영(Seoyoung Kim),황순욱(Soonwook Hwang) Korean Institute of Information Scientists and Eng 2016 정보과학회논문지 Vol.43 No.6

Hadoop-Spark 기반 설비 빅데이터 분석 플랫폼 구축

이태현(T. Lee),유은섭(E. S. Yu),문두환(D. Mun) Korean Society for Precision Engineering 2018 한국정밀공학회 학술발표대회 논문집 Vol.2018 No.10월

Towards Efficient Design and Implementation of a Hadoop-based Distributed Video Transcoding System in Cloud Computing Environment

Myoungjin Kim,Yun Cui,Seungho Han,Hanku Lee 보안공학연구지원센터 2013 International Journal of Multimedia and Ubiquitous Vol.8 No.2

A Study on Adaptive Smart Platform for Intelligent Software in Big Data Environment

김정식,김진홍 한국지식정보기술학회 2020 한국지식정보기술학회 논문지 Vol.15 No.3

Since smart platform convergence became a main issue of 4th industrial revolution, intelligent software-centric industry has carried out a policy from Internet of Things/Everything, Big Data, Artificial Intelligent, and Deep Learning. Now a day, this highest super technology provide user’s convenience, awareness, adaptation, and reactivity from intelligent software of IT. From their user-oriented service, we could make a great new leap forward to quality of software product. In addition, various convergence system of super national level is being made to stimulate both economy and industry. Nevertheless, it must be considered that the scale of these infrastructures is very large and that the conventional methods of inspecting infrastructures are very fast and time-consuming. These conventional methods are also dangerous for inspection team since the inspectors need to move or even climb on massive infrastructures to inspect places and areas that are difficult to reach. Besides, most conventional inspection methods are visual and the approaches applied are manual. Because of these problems mentioned above, adaptive smart platform and intelligent software have become important issues for users-aspect, industry-side, and Nation-oriented as time goes on. Accordingly, in this research paper, we propose adaptive smart platform that intelligent software have analyze, process, and refinement by big data. This platform is designed based on convergent software framework, such as Hadoop, HDFS and so on architectures utilizes our platform analyze massive data on their field by streamed data processing.