This project has moved and is read-only. For the latest updates, please go here.

Unmanaged memory use with Managed Esent / PersistentDictionary

Aug 25, 2015 at 3:12 AM
When starting on our project we stored records of items in a plain dictionary. Something like
Dictionary<Guid, List<Record>> records;
Pretty simple stuff. However this has an obvious flaw. Our product runs as a windows service and is expected to run for a long time. The number of records grows as the service runs, so this amounts to a basically a memory leak. Rather than wait for our database team to make changes for us we decided to try using PersistentDictionary to store the records. They are stored in PersistentDictionary in a format something like this:
PersistentDictionary<string, StoredRecord> records

struct StoredRecord {
    public Guid Id;
    public string SerializeRecords;
}
While this is not perfect it will do for the time being and it was easy to implement. However I monitored our memory use and found results that surprised me. I expected our memory usage to be lower on average and more stable since most data is inside the persistentdictionary, although there might be more allocations because we were serializing items going into and out of the persistent dictionary.

I compared memory usage before and after implementing the persistentdictionary storage using perf counters including the .NET memory performance counters, I found that 1) Memory usage measure by private bytes, is much higher with persistent dictionary (e.g. 150 MB vs 300 MB). 2) Memory usage as measure by private bytes still grows over time.

Looking closer at the .NET memory counters I can say that the reserved, committed, and bytes in all heaps are more stable over time with persisted dictionary storage. Private bytes is also significantly higher that any of these managed memory numbers. With a basic dictionary they are around the same values. This indicates to me that something in esent is using a significant chunk of memory and it is growing over time.

I wrote a small test program to read and write from persisted dictionary while I watched the memory perf counters. From what I can tell, it looks like private bytes stays constant as it writes to the persisted dictionary, however when it reads the data, private bytes will grow a bit every time.

So what is the caching behavior of esent/persisted dictionary? From my testing it seems like reads cause some data to be cached in unmanaged memory, while writes do not. Is there an easy way to verify my hypothesis? Can I manually clear any caches? Should I? or is caching all automatic? I'm just worried that I still have a memory leak and I can't tell.

Thanks.
Aug 25, 2015 at 9:50 PM
I think I found the way to control the unmanaged memory that is used. I adjusted the Database.Config.CacheSizeMax to more reasonable value for us. The default is 2e9 which seems to be way too large for what we need. Is this the correct way to do this?

Alternately I found I could just close the Persistent Dictionary and reopen, it however that added a lot of overhead to each read/write. I suspect this is not the "proper" way to use it.
Aug 25, 2015 at 11:39 PM
Correct, the Database.Config cache size is the most likely suspect. You can confirm by looking at perfmon.
-perfmon.msc
-Add counter
-Database
-Choose the 'Database Cache Memory Committed (MB)' and choose the correct Instance for your process
-Look at the usage.


-martin
Aug 25, 2015 at 11:48 PM
Oh yes, you could just set the CacheSizeMax to CacheSizeMin (PersistentDictionary's default is 64 MB), force it to dump some of the cached data, and then make it larger again.

There are two basic parts to the cached memory:
-Read cache. When shrinking the cache, this can just be thrown away. Read cache saves future reads from the disk.
-Write cache. When shrinking the cache, this has to be written to disk, so it may be slow. Write cache helps even out the writes that are done to the disk.

You can't control the ratio.

We do have intelligence in place to dynamically shrink the database cache if other processes on the system are using memory.

Hope that helps,


-martin
Aug 26, 2015 at 5:32 PM
Martin,

Thanks for the info. Adjusting CacheSizeMax/Min to fit our needs works fine. Those perf counters are also useful.

Thanks,

Mike
Jul 20, 2016 at 5:46 PM
Edited Jul 20, 2016 at 5:46 PM
How to set CacheSizeMax after the constructor of PersistentDictionary has been run?

Here is my scenario:
  • I have a code that uses PersistentDictionary() which will cause the database cache to be over 3GB.
  • After that code is run, I want to set CacheSizeMax to 100MB to force it to dump majority of the cached data.
  • I do NOT want to dispose the PersistentDictionary() object, because I am going to use this same object 10 minutes later.
The only way I can see about using CacheSizeMax is from the constructor like the following.
Is there a way to set it after using the dictionary without disposing it
        DatabaseConfig databaseConfig = new DatabaseConfig()
        {
            CacheSizeMax = 3 * 1024 * 1024 * 1024 / 8192;  // That is 3GB
        };

        var dictionary = new PersistentDictionary<TKey, string>("myfile", databaseConfig);
        // Use dictionary now...
        // How to set CacheSizeMax to 100MB after using the dictionary without disposing it?
Thanks.