Optimization Driven MapReduce Framework for Indexing and Retrieval of Big Data

( Hemn Barzan Abdalla ),( Awder Mohammed Ahmed ),( M. A. Al Sibahee ) 한국인터넷정보학회 2020 KSII Transactions on Internet and Information Syst Vol.14 No.5

With the technical advances, the amount of big data is increasing day-by-day such that the traditional software tools face a burden in handling them. Additionally, the presence of the imbalance data in big data is a massive concern to the research industry. In order to assure the effective management of big data and to deal with the imbalanced data, this paper proposes a new indexing algorithm for retrieving big data in the MapReduce framework. In mappers, the data clustering is done based on the Sparse Fuzzy-c-means (Sparse FCM) algorithm. The reducer combines the clusters generated by the mapper and again performs data clustering with the Sparse FCM algorithm. The two-level query matching is performed for determining the requested data. The first level query matching is performed for determining the cluster, and the second level query matching is done for accessing the requested data. The ranking of data is performed using the proposed Monarch chaotic whale optimization algorithm (M-CWOA), which is designed by combining Monarch butterfly optimization (MBO) [22] and chaotic whale optimization algorithm (CWOA) [21]. Here, the Parametric Enabled-Similarity Measure (PESM) is adapted for matching the similarities between two datasets. The proposed M-CWOA outperformed other methods with maximal precision of 0.9237, recall of 0.9371, F1-score of 0.9223, respectively.

Xpath에 의한 인터넷 문서의 레이아웃 추출 방법에 관한 연구

선복근,한광록 한국콘텐츠학회 2005 한국콘텐츠학회논문지 Vol.5 No.4

Currently most Internet documents including news data are made based on predefined templates, but templates are usually formed only for main data and are not helpful for information retrieval against indexes, advertisements, header data etc. Templates in such forms are not appropriate when Internet documents are used as data for information retrieval. In order to process Internet documents in various areas of information retrieval, it is necessary to detect additional information such as advertisements and page indexes.Thus this study proposes a method of detecting the layout of web pages by identifying the characteristics and structure of block tags that affect the layout of web pages and calculating distances between web pages. As a result of experiment, we can successfully extract 640 documents from 1000 samples and obtain 64% recall rate. This method is purposed to reduce the cost of web document automatic processing and improve its efficiency through applying the method to document preprocessing of information retrieval such as data extraction and document summarization. 현재 뉴스 데이터 등 대부분의 인터넷 문서는 일정한 템플릿을 기반으로 작성되고 있으며 템플릿은 메인 데이터 이외에 인덱스, 광고, 헤더데이터 등 정보검색에 도움이 되지 않는 형태로 구성되어 있다. 이는 인터넷 문서를 정보검색의 데이터로서 사용하려고 할 때 적합한 형태가 아니다. 그러므로 다양한 정보검색 분야에서 인터넷 문서를 처리하기 위해선 광고, 페이지 인덱스 등의 부가정보를 분별해야 한다.따라서 본 논문에서는 웹페이지의 레이아웃에 영향을 미치는 블럭 태그의 특징과 구조를 파악하고 웹페이지간의 거리를 계산하여, 웹페이지의 레이아웃을 검출하는 방법을 제안한다. 실험결과 1000개의 문서 중 640개를 분류했으며, 평균 64%의 recall 수치를 얻을 수 있었다. 이 방법을 데이터 추출, 문서요약 등의 정보검색 분야의 전처리 과정에 적용할 경우 문서의 자동화 처리 시간을 감소시키고 처리의 효율성을 높일 수 있을 것으로 기대된다.

A Multi-attribute Keyword Retrieval Mechanism for Encrypted Cloud Data

Ying Kong,Hui-juan Lu 보안공학연구지원센터 2016 International Journal of Security and Its Applicat Vol.10 No.12

With the development of cloud computing technology, more and more users have outsourced their data to cloud platform. To protect the safety of these data, some encryption methods are usually used. But it is very difficult question that how to effective retrieval and to use the encrypted data, we propose a multi-attribute keyword retrieval mechanism for encrypted cloud data. In this mechanism, we first define the local feature attributes of the keywords and the global feature attributes of the document. Then, we construct the mapping relationship between keywords and document attributes according to the basic principle of inverted index algorithm and establish the security index. Based on the above steps, we improve the weight of the key words and calculate the similarity score of the document by combining the multi-attribute score function. Experiments show that this mechanism not only can effectively improve the accuracy of the data retrieval, but also can greatly reduce the bandwidth consumption of system.

프라이버시를 보호하는 접근제어가 가능한 키워드 검색 기법

노건태(Geon Tae Noh),천지영(Ji Young Chun),정익래(Ik Rae Jeong),이동훈(Dong Hoon Lee) 한국정보보호학회 2009 정보보호학회논문지 Vol.19 No.5

민감한 개인정보를 보호하기 위해 데이터를 암호화하는 것은 필수적이다. 하지만 복호화 과정 없이 암호화된 데이터를 검색하기 위해서는 암호화된 데이터에서 검색이 가능한 효율적인 기법이 필요하다. 지금까지 수많은 검색 가능한 암호화 기법이 제안되었지만, 아직까지 이러한 기법들은 암호화된 데이터를 공유하기 위해 접근 권한을 갖고 있는 동적인 사용자(dynamic user)에 대해서 적합하지 않다. 기존의 검색 가능한 암호화 기법들에서는 특정 사용자(대칭키 환경에서의 데이터 제공자, 공개키 환경에서 데이터를 암호화한 공개키에 대응되는 비밀키를 갖고 있는 사용자)에 대해서만 암호화된 데이터에 접근이 가능하였다. 이러한 문제를 해결하기 위해 Stephen S. Yau 등은 데이터 공급자의 접근 정책에 따라서 사용자의 검색 능력을 제어할 수 있는 기법을 처음으로 제안하였다. 그러나 이 기법은 데이터 검색자의 프라이버시가 노출되는 문제점을 가진다. 따라서 본 논문에서는 이 기법의 문제점을 분석하고, 이러한 문제를 해결한 프라이버시를 보호하는 접근제어가 가능한 키워드 검색 기법을 제안한다. To protect sensitive personal information, data will be stored in encrypted form. However in order to retrieve these encrypted data without decryption, there need efficient search methods to enable the retrieval of the encrypted data. Until now, a number of searchable encryption schemes have been proposed but these schemes are not suitable when dynamic users who have the permission to access the data share the encrypted data. Since, in previous searchable encryption schemes, only specific user who is the data owner in symmetric key settings or has the secret key corresponding to the public key for the encrypted data in asymmetric key settings can access to the encrypted data. To solve this problem, Stephen S. Yau et al. firstly proposed the controlled privacy preserving keyword search scheme which can control the search capabilities of users according to access policies of the data provider. However, this scheme has the problem that the privacy of the data retrievers can be breached. In this paper, we firstly analyze the weakness of Stephen S. Yau et al.'s scheme and propose privacy preserving keyword search with access control. Our proposed scheme preserves the privacy of data retrievers.

전거데이터 기반 학술논문 검색시스템의 구축 및 평가에 관한 연구

이석형,곽승진 한국문헌정보학회 2012 한국문헌정보학회지 Vol.46 No.2

This study aims to develop and evaluate the authority data-based academic retrieval system that may provide search tools and additional information for search of academic papers with users. To evaluate the usability of authority data-based academic paper search system, this work studied 60 subjects who had experienced the academic paper service to measure their satisfaction on both search efficiency and system usability and additionally analyzed whether the difference between search efficiency and system availability is significant in the experimental search system among user groups. As a result, the authority data-based academic paper search system was analyzed to show a higher search efficiency and user satisfaction than the ordinary system. 본 연구의 목적은 학술논문 전거데이터 구축시스템을 통해 생성된 전거데이터를 기반으로 한 학술논문 검색시스템을 구축하고 검색 효율성과 시스템 유용성을 평가하는데 있다. 전거데이터 기반 학술논문 검색시스템은 저자 및 소속기관에 대한 전거데이터를 활용하여 질의어 자동완성, 검색결과 군집화, 저자네트워크 및 주제네트워크와 같은 부가 정보를 적용하였다. 본 연구에서 제안한 학술논문 검색시스템에 대하여 연구원, 도서관사서, 대학원생 이용자 집단 60명을 대상으로 검색 효율성과 시스템 유용성 평가를 수행하였다. 그 결과 전거데이터 기반 학술논문 검색시스템이 일반적인 학술논문 검색시스템에 비해 검색 효율성과 시스템 유용성이 높은 것으로 분석되었다.

디지털데이터를 통한 음악연구의 가능성

이미경 세계음악학회 2019 음악과 문화 Vol.0 No.41

This study attempted to show how digital data appeared in the field of musicology and what possibilities it has for music studies. To this end, I compared two sets of research cases that used analog data and the other using digital data. The first set to compare is the two cases where the performances on the record are the subjects of research, and the case of comparing and analyzing by the researcher's ear is compared with the case using the statistical method by converting the performances recorded on the record into digital data. The second set compares the work of A. Lomax, who collected over 1,800 songs, with recent research using big data analysis. Through the two sets of case comparisons, the research meanings, methodological limitations, and prospects of the method of studying music with digitized data are explored, and the possibility of such research methods in Korean music studies is discussed from the performance analysis perspective.

A Study on Effective Internet Data Extraction through Layout Detection

Sun Bok-Keun,Han Kwang-Rok The Korea Contents Association 2005 International Journal of Contents Vol.1 No.2

Currently most Internet documents including data are made based on predefined templates, but templates are usually formed only for main data and are not helpful for information retrieval against indexes, advertisements, header data etc. Templates in such forms are not appropriate when Internet documents are used as data for information retrieval. In order to process Internet documents in various areas of information retrieval, it is necessary to detect additional information such as advertisements and page indexes. Thus this study proposes a method of detecting the layout of Web pages by identifying the characteristics and structure of block tags that affect the layout of Web pages and calculating distances between Web pages. This method is purposed to reduce the cost of Web document automatic processing and improve processing efficiency by providing information about the structure of Web pages using templates through applying the method to information retrieval such as data extraction.

Exploration on the New Legal Discovery Model of China's Smart Judiciary

张光君,张翔 중국지역학회 2022 중국지역연구 Vol.9 No.3

In the process of promoting the upgrading of judicial intelligence, in order to realize the transformation of the effect of adjudication from “visible justice”―“expressing justice”― “acceptable justice”, the Supreme People’s Court of China proposed a new model of legal application based on the “retrieve system for similar cases” and supported by “brain-like intelligence technology”. However, due to the lack of innovation in legal methodology adapted to the big data of similar cases, judges still make legal discovery and “legal interpretation and reasoning” based on the small data. However, the wisdom of the legal community extracted on the basis of big data is difficult to have a substantial impact on the judge’s testimony from the level of legal methods. This makes the retrieval system of similar cases into a “chicken rib”. In order to fundamentally reverse the current problems of unclear methods of legal discovery and the ineffectiveness of the retrieval system for similar cases, it is urgent to stimulate the potential of big data technology. That is to say, through the transformation of the underlying thinking paradigm, a new mode of legal discovery is constructed, and then the efficiency of the retrieval system for similar cases is fully released. This paper adheres to the theory of “technology-institutional co-evolution” and follows the approach of “What (meaning of new legal discovery)-Why (value of new legal discovery)-How (implementation of new legal discovery)”. At the same time, this paper adopts the research methods such as the comparison of the principles of legal discovery in the two legal systems and the empirical analysis of the normative documents of the judiciary. The new model of legal discovery is based on the retrieval system of similar cases. After realizing the intelligent identification of similar cases with the help of big data, the “common factors” of similar cases are extracted through the comparison of types. Then, through the two-way communication between the abstract norm and the pending facts, the process realizes the finalization of specific specifications within the scope of the semantic range of the abstract norm. This model is conducive to unifying the standards for the application of the law and facilitates the same judgment in similar cases. In order to implement this model and improve the level of intellectualization in legal discovery, the main body of the retrieval report for similar cases should be changed from “the judge's individual combat” to “the coordinated attack of the judge + judge's assistant + lawyer”. After reducing unnecessary restrictions on starting conditions, the search is carried out for all valid cases. Next, the process of judging the similarity of similar cases is included in the presentation content of the retrieval report of similar cases. At the same time, the ranking of similar cases is based on “similarity of similar cases”, supplemented by “level of effectiveness”.

MPEG-7 기반의 멀티미디어 데이터 검색 시스템 설계

김경수 한국융합보안학회 2008 융합보안 논문지 Vol.8 No.4

멀티미디어 데이터의 급격한 양적 팽창은 원하는 데이터를 빠르고 정확하게 검색해야 한다는 새로운 과제를 안겨주었다. 이러한 효율적 검색을 위해 가장 중요한 기반이 되는 것이 바로 데 이터의 적절한 표준화이다. 2001년 국제 표준으로 제정된 MPEG-7은 바로 이러한 이유로 멀티 미디어 데이터의 표현에 대한 표준화를 다루고 있다. 본 논문에서 설계한 시스템은 MPEG-7에서 요구하는 내용기반 검색 방법인 하위 레벨 메타 데이터들을 이용한 유사성 검색과 상의 레벨 메타데이터들을 이용한 키워드 검색 기능을 제공 할 것이다. 또한, 상위 레벨 메타데이터와 하위 레벨 메타데이터들을 통합하여 검색하는 기능을 제공하여 사용자가 원하는 멀티미디어 정보를 보다 효율적으로 검색할 수 있도록 할 것이다. An increasing in quantity of multimedia data brought a new problem that expected data should be retrieved fast and exactly. The adequate representation is a key element for the efficient retrieval. For this reason, MPEG-7 standard was established for description of multimedia data in 2001. In this paper, we designed a Audio/Image Retrieval System based on MPEG-7 that can retrieve multimedia data like audio, image efficiently. And we integrated high-level and low-level schemas to retrieve datas for users.

주석정보와 특징정보를 이용한 비디오데이터 검색 시스템

이근왕(Lee, Keun-Wang) 한국산학기술학회 2006 한국산학기술학회논문지 Vol.7 No.6

본 논문에서는 대용량의 비디오 데이터에 대한 사용자의 다양한 의미검색을 지원하는 의미기반 비디오 검색 시스템을 제안한다. 제안하는 시스템은 주석기반 검색과 특정기반 검색을 통합한 에이전트를 이용하여 비디오 데이터의 내용 정보 추출 및 검색 과정을 자동으로 처리한다. 설계하고 구현한 시스템은 실험을 통한 성능 평가에서 비디오 데이터의 장면 검색에 대하여 기존 시스템 보다 재현율과 정확률의 증가를 보였다. In this thesis, we propose a semantics-based video retrieval system which supports semantics-retrieval for various users of massive video data. Proposed system automatically processes the extraction of contents information which video data has and retrieval process using agent which integrate annotation-based retrieval and feature-based retrieval. From experiment, the designed and implemented system shows increase of recall rate and precision rate for video data scene retrieval in performance assessment.