Multi-modality image fusion based on enhanced fuzzy radial basis function neural networks

Chao, Zhen,Kim, Dohyeon,Kim, Hee-Joung Elsevier 2018 Physica medica Vol.48 No.-

<P><B>Abstract</B></P> <P>In clinical applications, single modality images do not provide sufficient diagnostic information. Therefore, it is necessary to combine the advantages or complementarities of different modalities of images. Recently, neural network technique was applied to medical image fusion by many researchers, but there are still many deficiencies. In this study, we propose a novel fusion method to combine multi-modality medical images based on the enhanced fuzzy radial basis function neural network (Fuzzy-RBFNN), which includes five layers: input, fuzzy partition, front combination, inference, and output. Moreover, we propose a hybrid of the gravitational search algorithm (GSA) and error back propagation algorithm (EBPA) to train the network to update the parameters of the network. Two different patterns of images are used as inputs of the neural network, and the output is the fused image. A comparison with the conventional fusion methods and another neural network method through subjective observation and objective evaluation indexes reveals that the proposed method effectively synthesized the information of input images and achieved better results. Meanwhile, we also trained the network by using the EBPA and GSA, individually. The results reveal that the EBPGSA not only outperformed both EBPA and GSA, but also trained the neural network more accurately by analyzing the same evaluation indexes.</P> <P><B>Highlights</B></P> <P> <UL> <LI> We have developed conventional fuzzy radial basis function (Fuzzy-RBF) neural network. </LI> <LI> We proposed new method to train and update the neural network system. </LI> <LI> We have successfully applied improved fuzzy-RBF neural network to medical image fusion. </LI> </UL> </P>

Multi-Modal 감정인식 기반 서비스추론 기술 개발

고광은,심귀보 제어로봇시스템학회 2008 제어로봇시스템학회 국내학술대회 논문집 Vol.2008 No.10

In this paper, we develop the technique to infer service based on the user’s emotion state and needed to let them. As subjective cognition, the emotion is inclined to be impulsive and it unconsciously contains desire and intend of men. Emotion can be available as one of the context around user, and it is most include intend of user in the other context. For this reason, we separately develop the technique of emotion recognition based on user’s facial image and voice signal, and analyze the rate of recognition. We separate optimal feature based on facial image and voice signal to improve rate of emotion recognition in optional circumstance, and we implement the technique of Multi-Modal emotion recognition based on feature fusion by integration of each discrete emotion features. And then, we propose feasibility of service reasoning based on emotion recognition used with context setting scenario by using this result.

Multi-modal Medical Image Fusion Based on Non-subsampled Shearlet Transform

Xing Xiaoxue,Cao Fucheng,Shang Weiwei,Liu Fu 보안공학연구지원센터 2015 International Journal of Signal Processing, Image Vol.8 No.2

In order to provide more comprehensive and effective information for cancer diagnosis and tumor treatment planning, it is necessary to fuse multi-modal medical images, such as CT/MRI image, CT/PET image, MRI/SPECT image and so on. In this paper, a multi-modal medical image fusion method based on Non-subsampled Shearlet Transform is proposed. Firstly, in this method, source images are decomposed into low-pass and high-pass subbands by NSST. Then, due to the characteristic features--large sparsity and strong contrast, the high-frequency and low-frequency coefficients of the images are fused by the different fusion rules. Finally, the image is reconstructed by the inverse non-sampled shearlet transform. In the method, the fusion rules are designed based on the regional energy and the average gradient; the image entropy, relative quality, average gradient, standard deviation and spatial frequency were used to evaluate the fusion results objectively. In the experiments, CT and MRI images are chosen to verify the method. Both the visual and the objective analysis show that the proposed method is better than the conventional Wavelet-based and non-subsampled Contourlet-based methods.

Multi-modal Medical Image Fusion Based on the Multiwavelet and Nonsubsampled Direction Filter Bank

Peng Geng,Xing Su,Tan Xu,Jianshu Liu 보안공학연구지원센터 2015 International Journal of Signal Processing, Image Vol.8 No.11

Aiming at solving the fusion problem of multimodal medical images, a novel medical image fusion algorithm is present in this paper. The multiwavelet is combined with the NSDFB to construct the proposed Multi-NSDFB transform. The source images can be decomposed into several lowpass coefficient and highpass coefficient. And all coefficients can be decomposed into four direction subbands. The modified spatial frequency is adopted to motivate the pulse coupled neural network to select the every direction subbands coefficients. Experiment results demonstrate that the proposed algorithm can not only extract more important visual information from source images, but also effectively avoid the introduction of artificial information. The present scheme outperforms the redundant discrete wavelet transform-based, and the Ripplet transform-based in terms of both visual quality and objective evaluation.

Product of Likelihood Ratio Scores Fusion of Dynamic Face and On-line Signature Based Biometrics Verification Application Systems

Soltane Mohamed 보안공학연구지원센터 2015 International Journal of Database Theory and Appli Vol.8 No.4

In this paper, the use of finite Gaussian mixture modal (GMM) based Expectation Maximization (EM) estimated algorithm for score level data fusion is proposed. Automated biometric systems for human identification measure a “signature” of the human body, compare the resulting characteristic to a database, and render an application dependent decision. These biometric systems for personal authentication and identification are based upon physiological or behavioral features which are typically distinctive, Multi-biometric systems, which consolidate information from multiple biometric sources, are gaining popularity because they are able to overcome limitations such as non-universality, noisy sensor data, large intra-user variations and susceptibility to spoof attacks that are commonly encountered in mono modal biometric systems. Simulation show that finite mixture modal (GMM) is quite effective in modelling the genuine and impostor score densities, fusion based the resulting density estimates achieves a significant performance on eNTERFACE 2005 multi-biometric database based on dynamic face and signature modalities.

Human Action Recognition Via Multi-modality Information

Zan Gao,Jian-ming Song,Hua Zhang,An-An Liu,Yan-bing Xue,Guang-ping Xu 대한전기학회 2014 Journal of Electrical Engineering & Technology Vol.9 No.2

In this paper, we propose pyramid appearance and global structure action descriptors on both RGB and depth motion history images and a model-free method for human action recognition. In proposed algorithm, we firstly construct motion history image for both RGB and depth channels, at the same time, depth information is employed to filter RGB information, after that, different action descriptors are extracted from depth and RGB MHIs to represent these actions, and then multimodality information collaborative representation and recognition model, in which multi-modality information are put into object function naturally, and information fusion and action recognition also be done together, is proposed to classify human actions. To demonstrate the superiority of the proposed method, we evaluate it on MSR Action3D and DHA datasets, the well-known dataset for human action recognition. Large scale experiment shows our descriptors are robust, stable and efficient, when comparing with the-state-of-the-art algorithms, the performances of our descriptors are better than that of them, further, the performance of combined descriptors is much better than just using sole descriptor. What is more, our proposed model outperforms the state-of-the-art methods on both MSR Action3D and DHA datasets.

다중 센서 융합 알고리즘을 이용한 운전자의 감정 및 주의력 인식 기술 개발

한철훈(Cheol-Hun Han),심귀보(Kwee-Bo Sim) 한국지능시스템학회 2008 한국지능시스템학회논문지 Vol.18 No.6

최근 자동차 산업 및 기술이 발전함에 띠라 기계적인 부분에서 서비스적인 부분으로 관심이 점점 바뀌고 있는 추세이다. 이와 같은 추세에 발맞추어 운전자에게 보다 안정적이며 편리한 운전 환경을 조성하기 위한 방법으로 감정 및 인지 인식에 대한 관심이 점점 높아지고 있다. 감정 및 주의력을 인식하는 것은 감정공학 기술로서 이 기술은 1980년대 후반부터 얼굴, 음성, 제스처를 통해 인간의 감정을 분석하고 이를 통해 인간 진화적인 서비스를 제공하기 위한 기술로 연구되어 왔다. 이와 같은 기술을 자동차 기술에 접목시키고 운전자의 안정적인 주행을 돕고 운전자의 감정 및 인지 상황에 따른 다양한 서비스를 제공할 수 있다. 또한 Real-Time으로 운전자의 제스처를 인식하여 졸음운전이나 부주의에 의한 사고를 사전에 예방하고 보다 안전한 운전을 돕는 서비스가 필요시 되고 있다. 본 논문은 운전자가 안전 운전을 하기 위해 생체-행동 신호를 이용하여 감정 및 졸음, 주의력의 신호를 추출하여 일정한 형태의 데이터베이스로 구축하고, 구축된 데이터를 이용하여 운전자의 감정 및 졸음, 주의력의 특징 점들을 검출하여, 그 결과 값을 Multi-Modal 방법을 통해 융합함으로써 운전자의 감정 및 주의력 상태를 인식할 수 있는 시스템을 개발하는데 목표를 두고 있다. As the automobile industry and technologies are developed, driver's tend to more concern about service matters than mechanical matters. For this reason, interests about recognition of human knowledge and emotion to make safe and convenient driving environment for driver are increasing more and more. recognition of human knowledge and emotion era emotion engineering technology which has been studied since the late 1980s to provide people with human-friendly services. Emotion engineering technology analyzes people's emotion through their faces, voices and gestures, so if we use this technology for automobile, we can supply drivers with various kinds of service for each driver's situation and help them drive safely. Furthermore, we can prevent accidents which are caused by careless driving or dozing off while driving by recognizing driver's gestures. the purpose of this paper is to develop a system which can recognize states of driver's emotion and attention for safe driving. First of ail, we detect a signals of driver's emotion by using bio-motion signals, sleepiness and attention, and then we build several types of databases. by analyzing this databases, we find some special features about drivers' emotion, sleepiness and attention, and fuse the results through Multi-Modal mothod so that it is possible to develop the system.

Human Action Recognition Via Multi-modality Information

Gao, Zan,Song, Jian-Ming,Zhang, Hua,Liu, An-An,Xue, Yan-Bing,Xu, Guang-Ping The Korean Institute of Electrical Engineers 2014 Journal of Electrical Engineering & Technology Vol.9 No.2

In this paper, we propose pyramid appearance and global structure action descriptors on both RGB and depth motion history images and a model-free method for human action recognition. In proposed algorithm, we firstly construct motion history image for both RGB and depth channels, at the same time, depth information is employed to filter RGB information, after that, different action descriptors are extracted from depth and RGB MHIs to represent these actions, and then multimodality information collaborative representation and recognition model, in which multi-modality information are put into object function naturally, and information fusion and action recognition also be done together, is proposed to classify human actions. To demonstrate the superiority of the proposed method, we evaluate it on MSR Action3D and DHA datasets, the well-known dataset for human action recognition. Large scale experiment shows our descriptors are robust, stable and efficient, when comparing with the-state-of-the-art algorithms, the performances of our descriptors are better than that of them, further, the performance of combined descriptors is much better than just using sole descriptor. What is more, our proposed model outperforms the state-of-the-art methods on both MSR Action3D and DHA datasets.

Enhancing Recommender Systems by Fusing Diverse Information Sources through Data Transformation and Feature Selection

( Thi-linh Ho ),( Anh-cuong Le ),( Dinh-hong Vu ) 한국인터넷정보학회 2023 KSII Transactions on Internet and Information Syst Vol.17 No.5

Recommender systems aim to recommend items to users by taking into account their probable interests. This study focuses on creating a model that utilizes multiple sources of information about users and items by employing a multimodality approach. The study addresses the task of how to gather information from different sources (modalities) and transform them into a uniform format, resulting in a multi-modal feature description for users and items. This work also aims to transform and represent the features extracted from different modalities so that the information is in a compatible format for integration and contains important, useful information for the prediction model. To achieve this goal, we propose a novel multi-modal recommendation model, which involves extracting latent features of users and items from a utility matrix using matrix factorization techniques. Various transformation techniques are utilized to extract features from other sources of information such as user reviews, item descriptions, and item categories. We also proposed the use of Principal Component Analysis (PCA) and Feature Selection techniques to reduce the data dimension and extract important features as well as remove noisy features to increase the accuracy of the model. We conducted several different experimental models based on different subsets of modalities on the MovieLens and Amazon sub-category datasets. According to the experimental results, the proposed model significantly enhances the accuracy of recommendations when compared to SVD, which is acknowledged as one of the most effective models for recommender systems. Specifically, the proposed model reduces the RMSE by a range of 4.8% to 21.43% and increases the Precision by a range of 2.07% to 26.49% for the Amazon datasets. Similarly, for the MovieLens dataset, the proposed model reduces the RMSE by 45.61% and increases the Precision by 14.06%. Additionally, the experimental results on both datasets demonstrate that combining information from multiple modalities in the proposed model leads to superior outcomes compared to relying on a single type of information.

Deep Robotic Grasping Prediction with Hierarchical RGB-D Fusion

Yaoxian Song,Jun Wen,Dongfang Liu,Changbin Yu 제어·로봇·시스템학회 2022 International Journal of Control, Automation, and Vol.20 No.1

Vision-based robotic grasping is a fundamental task in robotic control. Dexterous and precise grasp control of the robotic arm is challenging and a critical technique for the manufacturing and emerging robot service industry. Current state-of-art methods adopt RGB-D images or point clouds in an attempt to obtain an accurate, robust, and real-time policy. However, most of these methods only use single modal data or ignore the uncertainty of sampling data especially the depth information. Even they leverage multi-modal data, they seldom fuse the features in different scales. All of these results in unreliable grasp prediction inevitably. In this paper, we propose a novel multi-modal neural network to predict grasps in real-time. The key idea is to fuse RGB and depth information hierarchically and quantify the uncertainty of raw depth data to re-weight the depth features. For higher grasping performance, a background extraction module and depth re-estimation module are used to reduce the influence caused by the incompletion and low-quality of the raw data. We evaluate the performance on the Cornell Grasp Dataset and provide a series of extensive experiments to demonstrate the advantages of our method on a real robot. The results indicate the superiority of our proposed method by outperforming the state-of-the-art methods significantly in all metrics.