Imaginary Soundscape 기반의 딥러닝을 활용한 회화와 음악의 매칭 및 다중 감각을 이용한 융합적 평가 방법 = Convergence evaluation method using multisensory and matching painting and music using deep learning based on imaginary soundscape

다국어 초록 (Multilingual Abstract)

In this study, we introduced the technique of matching classical music using deep learning to design soundscape that can help the viewer appreciate painting and proposed an evaluation index to evaluate how well matching painting and music. The evaluation index was conducted with suitability evaluation through the Likeard 5-point scale and evaluation in a multimodal aspect. The suitability evaluation score of the 13 test participants for the deep learning based best match between painting and music was 3.74/5.0 and band the average cosine similarity of the multimodal evaluation of 13 participants was 0.79. We expect multimodal evaluation to be an evaluation index that can measure a new user experience. In addition, this study aims to improve the experience of multisensory artworks by proposing the interaction between visual and auditory. The proposed matching of painting and music method can be used in multisensory artwork exhibition and furthermore it will increase the accessibility of visually impaired people to appreciate artworks.

국문 초록 (Abstract)

본 연구에서는 회화 감상에 도움이 되는 사운드스케이프를 구성하기 위해 딥러닝 기술을 활용하여 클래식 음악을 매칭하는 기술을 소개하고 회화와 음악 매칭이 얼마나 잘 되었는지에 대해 평가할 수 있는 평가 지표를 제안한다. 평가 지표는 리커드 5점 척도를 통한 적합도 평가와 멀티모달 측면의 평가로 진행하였다. 회화와 음악 매칭에 대해 13명의 실험 참가자의 적합도 평가의 점수는 3.74/5.0 이었고, 또한 13명의 실험 참가자의 멀티모달 평가에서 회화와 음악 매칭의 코사인 유사도의 평균은 0.79였다. 멀티모달적 평가는 새로운 사용자 경험을 측정할 수 있는 평가 지표가 될 것으로 기대된다. 또한 본 연구를 통해 시각과 청각의 인터랙션을 제안함으로써 다중감각 예술작품 경험을 향상시키고자 하였다. 본 연구에서 제안된 회화와 음악 매칭이 다중감각 예술작품 전시에서 활용되며 더 나아가 이는 시각 장애인들의 예술작품 감상에 대한 접근성을 높일 수 있을 것이라 기대한다.

참고문헌 (Reference)

1 정현원, "감성 평가를 위한 감성의 의미 재정립과 어휘 체계에 관한 연구" 대한인간공학회 26 (26): 17-25, 2007

2 "https://www.bunkerdelumieres.com"

3 S. J. Shin., "classification of adjectives" 6 : 19-40, 1999

4 M. Muller-Eberstein, "Translating Visual Art into Music" 2019

5 R. Murray Schafer, "The Soundscape : Our Sonic Environment and the Tuning of the World" Destiny Books 1977

6 R. E. Cytowic, "Synesthesia : A union of the senses" MIT press 2002

7 Y. Aytar, "Soundnet:Learning sound representations from unlabeled video" 892-900, 2016

8 D. B. Faustino, "SensArt demo: A multisensory prototype for engaging with visual art" 462-465, 2017

9 A. Howard, "Searching for mobilenetv3" 1314-1324, 2019

10 A. Sharghi, "Query-focused video summarization: Dataset, evaluation, and a memory network based approach" 4788-4797, 2017

11 Chi Thanh Vi, "Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition" Elsevier BV 108 : 1-14, 2017

12 Siyi Wang, "Museum as a Sensory Space: A Discussion of Communication Effect of Multi-Senses in Taizhou Museum" MDPI AG 12 (12): 3061-, 2020

13 R. Baeza-Yates, "Modern information retrieval(Vol. 463)" ACM press 1999

14 Y. Kajihara, "Imaginary Soundscape : Cross-Modal Approach to Generate Pseudo Sound Environments" 2017

15 Multisensory Artworks exhibition, "Human ICT Convergence, (professor: Jun Dong Cho) BlindTouch (Multisensory Painting Platform for the Blind)Exhibition; Exhibition Place" Siloam Center for the Blind S-Gallery 2019

16 Thomas Baumgartner, "From emotion perception to emotion experience: Emotions evoked by pictures and classical music" Elsevier BV 60 (60): 34-43, 2006

17 J. Iranzo Bartolome, "Exploring Art with a Voice Controlled Multimodal Guide for Blind People" 383-390, 2019

18 K. W. Guk., "Examples of applications by AI technology and industry sectors" 20 : 15-27, 2019

19 J. D. Cho, "Color Information Transfer Multi-modal Interface Concept Design for People with Visually Impairment to Appreciate Works of Art-Focused on the Case of"Blind-Touch", a Reproduction Art for Blind-" 2 (2): 44-58, 2019

20 L. Cavazos Quero, "An Interactive Multimodal Guide to Improve Art Accessibility for Blind People" 346-348, 2018

21 Y. G. Jeon, "(A)Study on the Sound Uses to Maximize Visual Images in Digital Media" Hansung University 2004