Institutional Repository
| Hippo: An enhancement of pipeline-aware in-memory caching for HDFS | |
| Wei, Lan (1); Lian, Wenbo (1); Liu, Kuien (1); Wang, Yongji (1); Wei, Lan | |
| 2014 | |
| Conference Name | 2014 23rd International Conference on Computer Communication and Networks, ICCCN 2014 |
| Conference Date | August 4, 2014 - August 7, 2014 |
| Conference Place | Shanghai, China |
| Indexed Type | EI |
| Publish Place | Institute of Electrical and Electronics Engineers Inc. |
| ISSN | 10952055 |
| ISBN | 9781479935727 |
| Department | (1) Institute of Software, Chinese Academy of Sciences, China |
| English Abstract | In the age of big data, distributed computing frameworks tend to coexist and collaborate in pipeline using one scheduler. While a variety of techniques for reducing I/O latency have been proposed, these are rarely specific for the whole pipeline performance. This paper proposes memory management logic called 'Hippo' which targets distributed systems and in particular 'pipelined' applications that might span differing big data frameworks. Though individual frameworks may have internal memory management primitives, Hippo proposes to make a generic framework that works agnostic of these highlevel operations. To increase the hit ratio of in-memory cache, this paper discusses the granularity of caching and how Hippo leverages the job dependency graph to make memory retention and pre-fetching decisions. Our evaluations demonstrate that job dependency is essential to improve the cache performance and a global cache policy maker, in most cases, significantly outperforms explicit caching by users.; In the age of big data, distributed computing frameworks tend to coexist and collaborate in pipeline using one scheduler. While a variety of techniques for reducing I/O latency have been proposed, these are rarely specific for the whole pipeline performance. This paper proposes memory management logic called 'Hippo' which targets distributed systems and in particular 'pipelined' applications that might span differing big data frameworks. Though individual frameworks may have internal memory management primitives, Hippo proposes to make a generic framework that works agnostic of these highlevel operations. To increase the hit ratio of in-memory cache, this paper discusses the granularity of caching and how Hippo leverages the job dependency graph to make memory retention and pre-fetching decisions. Our evaluations demonstrate that job dependency is essential to improve the cache performance and a global cache policy maker, in most cases, significantly outperforms explicit caching by users. |
| Language | 英语 |
| Content Type | 会议论文 |
| URI | http://ir.iscas.ac.cn/handle/311060/16610 |
| Collection | 中国科学院软件研究所 |
| Corresponding Author | Wei, Lan |
| Recommended Citation GB/T 7714 | Wei, Lan ,Lian, Wenbo ,Liu, Kuien ,et al. Hippo: An enhancement of pipeline-aware in-memory caching for HDFS[C]. Institute of Electrical and Electronics Engineers Inc.,2014. |
| Files in This Item: | There are no files associated with this item. | |||||
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment