An Efficient Log Data Processing Architecture for Internet Cloud Environments

다국어 초록 (Multilingual Abstract)

Big data management is becoming an increasingly important issue in both industry and academia of information science community today. One of the important categories of big data generated from software systems is log data. Log data is generally used for better services in various service providers and can also be used to improve system reliability. In this paper, we propose a novel big data management architecture specialized for log data. The proposed architecture provides a scalable log management system that consists of client and server side modules for efficient handling of log data. To support large and simultaneous log data from multiple clients, we adopt the Hadoop infrastructure in the server-side file system for storing and managing log data efficiently. We implement the proposed architecture to support various client environments and validate the efficiency through measurement studies. The results show that the proposed architecture performs better than the existing logging architecture by 42.8% on average. All components of the proposed architecture are implemented based on open source software and the developed prototypes are now publicly available.

참고문헌 (Reference)

1 S. Appel, "Towards benchmarking of AMQP" 99-100, 2010

2 S. Ghemawat, "The Google file system" 29-43, 2003

3 J.P. Lozi, "Remote core locking: migrating critical-section execution to improve the performance of multithreaded applications" Proceedings of the USENIX Annual Technical Conference (ATC) 2012

4 "RabbitMQ"

5 "Pig"

6 Yang Liu, "Multiple layer design for mass data transmission against channel congestion in IoT" Wiley-Blackwell 27 (27): 1126-1146, 2014

7 I-Ching Hsu, "Multilayer context cloud framework for mobile Web 2.0: a proposed infrastructure" Wiley-Blackwell 26 (26): 610-625, 2013

8 J. Zhao, "MapReduce: the programming model and practice" Tutorials of the ACM SIGMETRICS Conference 2009

9 "Log4j"

10 "Hive"

11 R. Fang, "High performance database logging using storage class memory" 2011

12 "Hadoop infrastructure"

13 "H-navi"

14 "Gartner, The future of the Internet"

15 M. Migliavacca, "DEFCON: high-performance event processing with information security" 2010

16 T. Steiner, "Crowdsourcing event detection in YouTube videos" 2011

17 J. Horey, "Big data platforms as a service: challenges and approach" Proceedings of the USENIX Workshop on Hot Topics in Cloud Computing (HotCloud) 2012

18 "Apache Flume"

19 S. Vinoski, "Advanced message queuing protocol" 10 (10): 87-89, 2006

20 B.P. Rimal, "A service-oriented taxonomical spectrum, cloudy challenges and opportunities of cloud computing" 25 (25): 796-819, 2012

21 Y. Lai, "A personalized mobile IPTV system with seamless video reconstruction algorithm in cloud networks" 24 (24): 1375-1387, 2011

22 X. Ye, "A novel blocks placement strategy for Hadoop" 3-7, 2012