Out of Memory(Microsoft.Isam.Esent.Interop.EsentOutOfMemoryException)

Apr 5, 2013 at 2:56 PM
Hi,

I am trying to use the persistence dictionary in one of my project. The project require to create more than 1000 persistence dictionary to store the different data structure and segments. I was able to create all 1000 dictionaries and query in memory, but when I start the application and trying to create the dictionaries from the file, it always crash and throw Out of Memory when it reach to 188-190 dictionary.
Below is a sample code to test and the stuck trace. I am testing this in my window 8 operating system.
        try
        {
           PersistentDictionary<string,string> Data;

           List<PersistentDictionary<string, string>> dd = new List<PersistentDictionary<string, string>>();

            for (int i = 0; i < 500; i++)
            {
             Data = new PersistentDictionary<string,string>( Path.Combine(@"C:\Repository", i.ToString()));
             dd.Add(Data);
            }

        }
        catch (Exception ex)
        {

            string e = ex.Message;
        }
at Microsoft.Isam.Esent.Interop.Api.Check(Int32 err) in e:\src\codeplex_svn\codeplex\EsentInterop\Api.cs:line 2856
at Microsoft.Isam.Esent.Interop.Api.JetAttachDatabase(JET_SESID sesid, String database, AttachDatabaseGrbit grbit) in e:\src\codeplex_svn\codeplex\EsentInterop\Api.cs:line 371
at Microsoft.Isam.Esent.Collections.Generic.PersistentDictionary2.CheckDatabaseMetaData(String database) in e:\src\codeplex_svn\codeplex\EsentCollections\PersistentDictionary.cs:line 745
at Microsoft.Isam.Esent.Collections.Generic.PersistentDictionary
2..ctor(String directory) in e:\src\codeplex_svn\codeplex\EsentCollections\PersistentDictionary.cs:line 154
Apr 7, 2013 at 1:30 AM
Please advice, this is in the critical path of my project and need to resolve this issue to progress.
Apr 7, 2013 at 12:54 PM
You're misusing ESENT. One database is enough. You're trying to create 500 databases. It won't work.

If you need 500 dictionaries, you may need either 500 tables in a single DB, or one table with complex primary key. Which ons of those solutions is better depends on the write parrerns.

Using my esentserialize library, here's the record class for you. It implements approach #2, with the single table. It implies both key and value are relatively short (no more 127 characters) unicode strings, if it's not the case, [EseText] must be altered with parameters, or replaced with [EseLongText].
[EseTable( "dictionaries" )]
[EsePrimaryIndex( "id", "+nDictionary\0+key\0\0" )] //< Note how the primary index is composed: it's dictionary #, followed by the key
public class DictionaryEntry
{
    [EseShortInt]
    private short nDictionary;  // The dictionary #

    [EseText]
    public string key { get; private set; }

    [EseText]
    public string value { get; set; }
}
Sample code implementing a dictionary above my library:
https://esentserialize.codeplex.com/SourceControl/changeset/view/a03b66824099b51a65194a59598a6ce78eda2a67#DictionaryDemo/Dictionary/DemoDictionary.cs

Using similar approach, you can easily implement 1000 persistent dictionaries above this single table.
Apr 9, 2013 at 6:04 PM
Thank you for the direction. I will look at esentserialize library.
Coordinator
Apr 9, 2013 at 9:23 PM
I haven't had the time to try it out myself.
ESE should be able to run with 1024 instances, but that is a hard-coded maximum. There are other resource constraints that may be coming i to play. For instance: Are you running in a 32-bit or 64-bit process? It wouldn't surprise me to run out of memory fairly quickly in a 32-bit process. 2/4 GB is a lot smaller now than it used to be!

But having 500 instances on a single disk may come up with other scalability issues. Even if it's functional, the performance may be questionable.

-martin