Memory leak in ManagedEsent?

Oct 17, 2011 at 5:25 PM
Edited Oct 17, 2011 at 6:15 PM

Hello.

Thanks for producing this api lib, very helpful.

But I experienced memory issue while I am trying to use it.

What I am doing is fairly simple. I followed sample code, but I put a very large data (49KB) in every row and produce 2000 rows in total. 

The memory usage increases while more number of rows are inserted. If I insert like 100,000 rows, the program will eat up 1G memory and dies.

Here is the code (please ignore the string data. I just copied part of today's techcrunch news for testing. actually I cut it in this post just for viewing conveniences. In my testing code, the string is 49KB long.)

 

public static void test()
        {
            string techcrunchString = @"The Latest from TechCrunch 
CMU Researchers Turn Any Surface Into A Touchscreen
Web Design Community Treehouse Raises $600K From Reid Hoffman, Kevin Rose, And Others
Greylock Looks To Help Portfolio Companies Recruit Talent With New Hires
UberMedia Quietly (Inadvertently?) Releases Chime.in, A Mobile Social Networking App
T-Mobile Announces The Dual-Screen LG DoublePlay, Launching November 2nd?
Watch An iPhone 4S and Samsung Galaxy S II Take Three Nasty Drops Onto Concrete
Facebook, NRDC & Opower To Partner On Energy-Saving Social App
CTIA�s New Alert Guidelines Could Mean The End Of �Bill Shock�
Grockit Gets A $7 Million Venture Infusion And Launches Video Q&A Site Grockit Answers
Gorgeous Photos, Tablet Browsing: 500px Debuts New iPad App
Samsung Galaxy Nexus, HTC Vigor To Launch November 10?
Freelancer.com: Facebook App, 3D, HTML5, And Cocoa Jobs On The Rise
iPhone 4S First Weekend Sales Exceeds 4 Million, Doubles The Pace Of The iPhone 4
Wahanda Secures $5.5 Million From Fidelity Growth Partners Europe
Look Out Uber: GroundLink Launches An Affordable, Mobile Private Car Service For New Yorkers
Video Collaboration Software Maker ViVu Acquired By Polycom
With 400,000 Users Under Its Belt, SohoOS Plans Major Revamp
5 Product Innovations From CEATEC 2011 In Japan (Video Gallery)
Digital Media Companies Inuvo And Vertro To Merge
RIM Apologizes With Free Apps & Technical Support For Three Days Of Downtime
CMU Researchers Turn Any Surface Into A Touchscreen
Posted: 17 Oct 2011 09:14 AM PDT
....";

            JET_INSTANCE instance;
            JET_SESID sesid;
            JET_DBID dbid;
            JET_TABLEID tableid;

            JET_COLUMNDEF columndef = new JET_COLUMNDEF();

            // Initialize ESENT. Setting JET_param.CircularLog to 1 means ESENT will automatically
            // delete unneeded logfiles. JetInit will inspect the logfiles to see if the last
            // shutdown was clean. If it wasn't (e.g. the application crashed) recovery will be
            // run automatically bringing the database to a consistent state.
            Api.JetCreateInstance(out instance, "instance");
            Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null);
            Api.JetInit(ref instance);
            Api.JetBeginSession(instance, out sesid, null, null);

            // Create the database. To open an existing database use the JetAttachDatabase and 
            // JetOpenDatabase APIs.
            Api.JetCreateDatabase(sesid, "edbtest.db", null, out dbid, CreateDatabaseGrbit.OverwriteExisting);

            // Create the table. Meta-data operations are transacted and can be performed concurrently.
            // For example, one session can add a column to a table while another session is reading
            // or updating records in the same table.
            // This table has no indexes defined, so it will use the default sequential index. Indexes
            // can be defined with the JetCreateIndex API.
            Api.JetBeginTransaction(sesid);
            Api.JetCreateTable(sesid, dbid, "table", 0, 100, out tableid);

            JET_COLUMNID id;
            columndef.coltyp = JET_coltyp.Binary;
            columndef.cp = JET_CP.ASCII;
            Api.JetAddColumn(sesid, tableid, "id", columndef, null, 0, out id);

            JET_COLUMNID blob;
            columndef.coltyp = JET_coltyp.LongBinary;
            //columndef.cp = JET_CP.ASCII;
            Api.JetAddColumn(sesid, tableid, "blob", columndef, null, 0, out blob);

            string indexDef = "+id\0\0";
            Api.JetCreateIndex(sesid, tableid, "primary", CreateIndexGrbit.IndexPrimary, indexDef, indexDef.Length, 100);
            //Api.JetSetCurrentIndex(sesid, tableid, null);
            Api.JetCommitTransaction(sesid, CommitTransactionGrbit.LazyFlush);

            long Process_MemoryStart = 0;
            Process MyProcess = System.Diagnostics.Process.GetCurrentProcess();
            Process_MemoryStart = MyProcess.PrivateMemorySize64;
            Console.WriteLine("Before loop : " + Process_MemoryStart / 1024 + "KB");

            int i = 0;
            for (int t = 0; t < 5; t++)
            {
                Api.JetBeginTransaction(sesid);
                for (int j = 0; j < 500; j++)
                {
                    i = t * 500 + j;
                    string dataString = techcrunchString + i.ToString();

                    byte[] data = Encoding.UTF8.GetBytes(dataString);
                    string keyString = i.ToString();
                    byte[] key = Encoding.UTF8.GetBytes(keyString);

                    //store
                    Api.MakeKey(sesid, tableid, key, MakeKeyGrbit.NewKey);
                    bool exists = Api.TrySeek(sesid, tableid, SeekGrbit.SeekEQ);
                    
                    if (exists)
                    {
                        Api.JetPrepareUpdate(sesid, tableid, JET_prep.ReplaceNoLock);
                        //Console.WriteLine("store: " + "update");
                    }
                    else
                    {
                        Api.JetPrepareUpdate(sesid, tableid, JET_prep.Insert);
                        Api.SetColumn(sesid, tableid, id, key);
                        //Console.WriteLine("store: " + "insert");
                    }
                    Api.SetColumn(sesid, tableid, blob, data);
                    Api.JetUpdate(sesid, tableid);

                    if (i % 500 == 0)
                    {
                        long Process_MemoryStart1 = 0;
                        Process MyProcess1 = System.Diagnostics.Process.GetCurrentProcess();
                        Process_MemoryStart1 = MyProcess1.PrivateMemorySize64;
                        Console.WriteLine("Finished " + i.ToString() + " : " + Process_MemoryStart1 / 1024 + "KB");
                        
                    }

                }
                Api.JetCommitTransaction(sesid, CommitTransactionGrbit.None);
                

            }
            Process_MemoryStart = 0;
            MyProcess = System.Diagnostics.Process.GetCurrentProcess();
            Process_MemoryStart = MyProcess.PrivateMemorySize64;
            Console.WriteLine("Loop finished: " + Process_MemoryStart / 1024 + "KB");

            // Terminate ESENT. This performs a clean shutdown.
            Api.JetCloseTable(sesid, tableid);
            Process_MemoryStart = 0;
            MyProcess = System.Diagnostics.Process.GetCurrentProcess();
            Process_MemoryStart = MyProcess.PrivateMemorySize64;
            Console.WriteLine("After close table: " + Process_MemoryStart / 1024 + "KB");

            Api.JetEndSession(sesid, EndSessionGrbit.None);
            Process_MemoryStart = 0;
            MyProcess = System.Diagnostics.Process.GetCurrentProcess();
            Process_MemoryStart = MyProcess.PrivateMemorySize64;
            Console.WriteLine("After end session: " + Process_MemoryStart / 1024 + "KB");

            Api.JetTerm(instance);
            
            Process_MemoryStart = 0;
            MyProcess = System.Diagnostics.Process.GetCurrentProcess();
            Process_MemoryStart = MyProcess.PrivateMemorySize64;
            Console.WriteLine("After term instance: " + Process_MemoryStart / 1024 + "KB");
        }

 

 

 

In above code, It goes up to about 100 MB. Only if I do Api.JetTerm(instance), the memory is freed. 

In my real problem, I have to constantly insert rows of large amount of data for many many times, so it won't work for me in this way as the memory will be eventually eaten up.

Could you please help me with this? Why esent holds the memory even if I commited the transaction? I am suspecting it is the undo things inside esent which holds the memory, and if it is, how to turn it off? I don't need undo thing.

Thanks

Oct 17, 2011 at 6:14 PM

by the way, I have tried to set CacheSizeMax = 1024, hinted from this web page http://www.nikosbaxevanis.com/bonus-bits/2010/10/adventures-using-rhino-servicebus.html

But it does not help. I also tried to it to 512, no help.

I also played around with MaxVerPages, StartFlushThreshold, StopFlushThreshold, CacheSize, CacheSizeMin, etc, no effects on this problem at all.

Thanks

Oct 17, 2011 at 7:11 PM

It is solved.

I took a wrong way to set CacheSizeMax. 

The correct way is

Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CacheSizeMax, 1024, null);

 

Now it is working and memory not increasing

Thanks