멀티코어 환경을 위한 강화학습 기반 캐시 파티셔닝 = A Reinforcement Learning-Based Cache Partitioning Scheme for Multi-Core Environments

다국어 초록 (Multilingual Abstract)

Most processors currently in use provide the shared last-level cache (LLC). Therefore, when multiple applications compete intensely for the LLC, its hit ratio is adversely impacted by extremely frequent cache line replacement; this may result in a significant degradation of the overall performance. The hardware-based cache partitioning techniques can relieve this issue by isolating the cache space of a core from others. However, it is necessary to use an adaptive and intelligent cache partitioning algorithm to dynamically determine the optimal cache partition. Reinforcement learning is an appropriate approach for this kind of problems. However, its model complexity skyrockets as the number of the applications to partition increases. This paper proposes a reinforcement learning-based cache partitioning scheme that can support a large number of running applications. Firstly, we built a reinforcement learning model and made it learn to perform cache partitioning for a small number of applications. We then extended it by clustering applications with the information obtained via supervised learning models for cache-use characteristic predictions, which enabled cache partitioning for more applications and resulted in performance gains of up to 19.75%.

국문 초록 (Abstract)

많은 상용 프로세서들은 LLC(last-level cache)를 모든 코어들이 공유하여 사용하기 때문에 LLC가 병목이 되어 전체 시스템 성능이 저하된다. 하드웨어 캐시 파티셔닝 기법을 적용하여 이런 문제를 해결할 수 있지만, 적절한 캐시 파티션을 결정하는 것은 지능적인 알고리즘을 필요로 한다. 강화학습을 사용한 캐시 파티셔닝은 애플리케이션의 수가 증가함에 따라 모델 복잡도가 폭발적으로 증가한다. 본 논문은 캐시 파티셔닝 문제의 강화학습 적용을 위해 모델 복잡도의 폭발적 증가를 억제하고 다중 애플리케이션에 대해 확장 가능한 강화학습 기반 캐시 파티셔닝을 제안한다. 먼저 적은 수의 애플리케이션에 대해 캐시 파티셔닝을 하는 강화학습 모델을 학습한다. 그 후 캐시 사용 특성 예측 지도학습 모델을 통해 얻은 정보로 애플리케이션들을 클러스터링하여 강화학습 모델을 확장 적용한다. 이를 통해 더 많은 애플리케이션에 대해서도 캐시 파티셔닝을 하여 최대 19.75%의 성능 향상을 얻을 수 있었다.

참고문헌 (Reference)

1 C. Xu, "dcat: Dynamic cache management for efficient, performance-sensitive infrastructure-as-a-service" 1-13, 2018

2 A. Mukkara, "Whirlpool: Improving dynamic cache management with static data classification" 44 (44): 113-127, 2016

3 X. Zhang, "Towards practicalpage coloring-based multicore cache management" 89-102, 2009

4 Y. Li, "Themis: Efficient and adaptive resource partitioning for reducing response delay in cloud gaming" 491-499, 2019

5 A. C. De Melo, "The new linux 'perf' tools" 18 : 1-42, 2010

6 J. L. Henning, "Spec cpu2006 benchmark descriptions" 34 (34): 1-17, 2006

7 T. Haarnoja, "Softactor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor" 1861-1870, 2018

8 R. S. Sutton, "Reinforcement learning: Anintroduction" MIT press 1998

9 J. Schulman, "Proximal policy optimization algorithms"

10 V. Mnih, "Playing atari with deep reinforcement learning"

11 G. Brockman, "Openai gym"

12 L. Zhang, "Lightweight dynamic partitioning for last level cache of multicore process or on real system" IEEE 33-38, 2012

13 N. El-Sayed, "Kpart: A hybrid cache partitioning-sharing technique for commodity multicores" IEEE 104-117, 2018

14 X. Hu, "Kinetic modeling of data eviction in cache" 351-364, 2016

15 I. Corporation, "Intel 64 and ia-32 architectures software developer’s manual, volume 3b: System programming guide, part 2, number 253669-032"

16 Z. Wang, "Dueling network architectures for deep rein-forcement learning" 1995-2003, 2016

17 H. Van Hasselt, "Deep reinforcementlearning with double q-learning" 30 : 2016

18 Y. Xiang, "Dcaps: dynamic cache allocation with partial sharing" 1-15, 2018

19 J. Park, "Copart: Coordinated partitioning of last-level cache and memory bandwidth for fairness-aware workload consolidation on commodity servers" 1-16, 2019

20 T. P. Lillicrap, "Continuous con-trol with deep reinforcement learning"

21 Y. Ye, "Coloris: a dynamiccache partitioning system using page coloring" IEEE 381-392, 2014

22 A. Herdrich, "Cache qos: From concept to realityin the intel® xeon® processor e5-2600 v3 product family" IEEE 657-668, 2016

23 M. Ester, "A density-based algorithm for discovering clusters in large spatialdatabases with noise" 96 : 226-231, 1996

24 R. Jain, "A coordinatedmulti-agent reinforcement learning approach to multi-levelcache co-partitioning" IEEE 800-805, 2017