심층 강화 학습을이용한 Luxo 캐릭터의 제어

국문 초록 (Abstract)

캐릭터로 하여금 시뮬레이션 내에서 사용자가 원하는 동작을 보이도록 물리 기반 제어기를만들 수 있다면 주변 환경의 변화와 다른 캐릭터와의 상호작용에 대하여 자연스러운 반응을 보이는 캐릭터 애니메이션을 생성할 수 있다. 최근 심층 강화학습을 이용해 물리 기반 제어기가 더 안정적이고 다양한 동작을 합성하도록 하는 연구가 다수 이루어져 왔다. 본 논문에서는다리가 하나 달린 픽사 애니메이션 스튜디오의 마스코트 캐릭터 Luxo를 주어진 목적지까지 뛰어 도착하게 하는 심층강화학습 모델을 제시한다. 효율적으로 뛰는 동작을 학습하도록 하기 위해서 Luxo의 각 관절의 각도값들을 선형 보간법으로 생성하여 참조 모션을 만들었으며, 캐릭터는 이를 모방하면서 균형을유지하여 목표한 위치까지 도달하도록 하는 제어 정책(control policy)을 학습한다. 참조 동작을 사용하지 않고 Luxo 동작을 제어하도록 학습된 정책과 비교한 실험 결과, 제안된 방법을 사용하면 사용자가 지정한 위치로 Luxo가 점프하며 이동하는 정책을 더 효율적으로 학습할 수 있었다.

다국어 초록 (Multilingual Abstract)

Motion synthesis using physics-based controllers can generate a character animation that interacts naturally with the given environment and other characters. Recently, various methods using deep neural networks have improved the quality of motions generated by physics-based controllers. In this paper, we present a control policy learned by deep reinforcement learning (DRL) that enables Luxo, the mascot character of Pixar animation studio, to run towards a random goal location while imitating a reference motion and maintaining its balance. Instead of directly training our DRL network to make Luxo reach a goal location, we use a reference motion that is generated to keep Luxo animation’s jumping style. The reference motion is generated by linearly interpolating predetermined poses, which are defined with Luxo character’s each joint angle. By applying our method, we could confirm a better Luxo policy compared to the one without any reference motions.

목차 (Table of Contents)

• 요약
• Abstract
• 1. 서론
• 2. 관련 연구
• 3. 캐릭터 및 참조 모션 생성
• 4. 심층 강화 학습
• 5. 실험 결과
• 6. 결론
• References

참고문헌 (Reference)

1 "stable baselines"

2 J. Schulman, "Trust region policy optimization" 37 : 1889-1897, 2015

3 S. Agrawal, "Task-based locomotion" 35 (35): 2016

4 D. Sharon, "Synthesis of controllers for stylized planar bipedal walking" 2387-2392, 2005

5 Jie Tan, "Stable Proportional-Derivative Controllers" Institute of Electrical and Electronics Engineers (IEEE) 31 (31): 34-44, 2011

6 Andrew Witkin, "Spacetime constraints" Association for Computing Machinery (ACM) 22 (22): 159-168, 1988

7 R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning" 8 (8): 229-256, 1992

8 K. Yin, "Simbicon : Simple biped locomotion control" 26 (26): 105-, 2007

9 R. Sutton, "Reinforcement Learning: An Introduction, ser. Adaptive Computation and Machine Learning series" MIT Press 1998

10 J. Schulman, "Proximal policy optimization algorithms" 2017

11 A. Sharma, "Phase-parametric policies for reinforcement learning in cyclic environments" 2018

12 J. M. Wang, "Optimizing walking controllers for uncertain inputs and environments" 2010

13 Yoonsang Lee, "Locomotion control for many-muscle humanoids" Association for Computing Machinery (ACM) 33 (33): 1-11, 2014

14 W. Yu, "Learning symmetry and low-energy locomotion"

15 S. Park, "Learning predictand-simulate policies from unorganized human motion data" 38 (38): 2019

16 X. B. Peng, "Learning locomotion skills using deeprl: Does the choice of action space matter?" 2017

17 A. Rajeswaran, "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations"

18 Kyungho Lee, "Interactive character animation by learning multi-objective control" Association for Computing Machinery (ACM) 37 (37): 1-10, 2019

19 J. Schulman, "High-dimensional continuous control using generalized advantage estimation" 2015

20 J. Z. Kolter, "Hierarchical apprenticeship learning, with application to quadruped locomotion" Curran Associates Inc. 769-776, 2007

21 K. Wampler, "Generalizing locomotion style to new animals with inverse optimal regression" 33 (33): 2014

22 S. Coros, "Generalized biped walking control" 29 (29): 130-, 2010

23 M. de Lasa, "Featurebased locomotion controllers" 29 (29): 2010

24 T. Kwon, "Fast and flexible multilegged locomotion using learned centroidal dynamics" 2020

25 N. M. O. Heess, "Emergence of locomotion behaviours in rich environments"

26 Igor Mordatch, "Discovery of complex behaviors through contact-invariant optimization" Association for Computing Machinery (ACM) 31 (31): 1-8, 2012

27 X. B. Peng, "Deepmimic: Example-guided deep reinforcement learning of physics-based character skills" 37 (37): 143:1-143:14, 2018

28 X. B. Peng, "Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning" 36 (36): 2017

29 Y. Lee, "Data-driven biped control" Association for Computing Machinery 2010

30 Kevin Bergamin, "DReCon" Association for Computing Machinery (ACM) 38 (38): 1-11, 2019

31 P. Abbeel, "Autonomous helicopter aerobatics through apprenticeship learning" 29 : 1608-1639, 2010

32 K. Yamane, "Animating nonhumanoid characters with human motion data" Eurographics Association 169-178, 2010

33 Jungdam Won, "Aerobatics control of flying creatures via self-regulated learning" Association for Computing Machinery (ACM) 37 (37): 1-10, 2019