From 638d23950716f3119057d725fbbf1590b807a849 Mon Sep 17 00:00:00 2001 From: Maysam Yabandeh Date: Mon, 16 Sep 2019 15:14:51 -0700 Subject: Charge block cache for cache internal usage (#5797) Summary: For our default block cache, each additional entry has extra memory overhead. It include LRUHandle (72 bytes currently) and the cache key (two varint64, file id and offset). The usage is not negligible. For example for block_size=4k, the overhead accounts for an extra 2% memory usage for the cache. The patch charging the cache for the extra usage, reducing untracked memory usage outside block cache. The feature is enabled by default and can be disabled by passing kDontChargeCacheMetadata to the cache constructor. This PR builds up on https://github.com/facebook/rocksdb/issues/4258 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5797 Test Plan: - Existing tests are updated to either disable the feature when the test has too much dependency on the old way of accounting the usage or increasing the cache capacity to account for the additional charge of metadata. - The Usage tests in cache_test.cc are augmented to test the cache usage under kFullChargeCacheMetadata. Differential Revision: D17396833 Pulled By: maysamyabandeh fbshipit-source-id: 7684ccb9f8a40ca595e4f5efcdb03623afea0c6f --- memtable/write_buffer_manager_test.cc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) (limited to 'memtable') diff --git a/memtable/write_buffer_manager_test.cc b/memtable/write_buffer_manager_test.cc index 06514eabd..23de06a62 100644 --- a/memtable/write_buffer_manager_test.cc +++ b/memtable/write_buffer_manager_test.cc @@ -51,8 +51,12 @@ TEST_F(WriteBufferManagerTest, ShouldFlush) { } TEST_F(WriteBufferManagerTest, CacheCost) { + LRUCacheOptions co; // 1GB cache - std::shared_ptr cache = NewLRUCache(1024 * 1024 * 1024, 4); + co.capacity = 1024 * 1024 * 1024; + co.num_shard_bits = 4; + co.metadata_charge_policy = kDontChargeCacheMetadata; + std::shared_ptr cache = NewLRUCache(co); // A write buffer manager of size 50MB std::unique_ptr wbf( new WriteBufferManager(50 * 1024 * 1024, cache)); -- cgit v1.2.3-70-g09d2