ManagedEsent Wiki & Documentation Rss Feed

New Comment on "PersistentDictionaryDocumentation"

Grimbly — Tue, 05 Mar 2013 23:19:08 GMT

I love this thing. It's every thing I needed and then some. Thanks for making this, you're awesome :)

New Comment on "Documentation"

hindermath — Thu, 05 Jan 2012 07:39:22 GMT

Thanx for this documentation and code snippets.

New Comment on "ManagedEsentSample"

hindermath — Wed, 04 Jan 2012 23:57:35 GMT

Cool, works as designed :-)! Fine work... much more coming in other database parts? Perfect NoSQL database without use of a comples server.

New Comment on "PersistentDictionaryDocumentation"

kpoxa — Mon, 11 Apr 2011 11:27:46 GMT

awesome!

Updated Wiki: StockSample

laurionb — Thu, 17 Feb 2011 20:08:05 GMT

ManagedEsent Stock Example

This sample code creates a database of stock prices and performs some basic queries and updates. This sample is included in the ManagedEsent release.

/// <summary>
/// Create a sample database containing some stock prices and
/// perform some basic queries.
/// </summary>
public static class StockSample
{
    /// <summary>
    /// Name of the table that will store the prices.
    /// </summary>
    private const string TableName = "stocks";

    /// <summary>
    /// Columnid of the stock symbol.
    /// </summary>
    private static JET_COLUMNID columnidSymbol;

    /// <summary>
    /// Columnid of the company name.
    /// </summary>
    private static JET_COLUMNID columnidName;

    /// <summary>
    /// Columnid of the stock price.
    /// </summary>
    private static JET_COLUMNID columnidPrice;

    /// <summary>
    /// Columnid of the number of shares owned.
    /// </summary>
    private static JET_COLUMNID columnidShares;

    /// <summary>
    /// Called on program startup.
    /// </summary>
    public static void Main()
    {
        Console.WriteLine();

        const string DatabaseName = "stocksample.edb";

        // First create the database. A real application would probably
        // check for the database first and only create it if needed.
        // Checking for the database can be done with File.Exists or
        // by calling JetAttachDatabase and seeing if a JET_ERR.DatabaseNotFound
        // error is thrown. (Check the Error property of the
        // EsentErrorException).
        CreateDatabase(DatabaseName);

        // Now the database has been created we can attach to it.
        // The Instance object we create is disposable, so the underlying ESENT
        // resource can be freed. The disposable objects wrapped around ESENT resources
        // _must_ be disposed properly. ESENT resources must be freed in a
        // specific order, which the finalizer doesn't necessarily respect.
        using (var instance = new Instance("stocksample"))
        {
            // Creating an Instance object doesn't call JetInit. 
            // This is done to allow some parameters to be set
            // before the instance is initialized.

            // Circular logging is very useful; it automatically deletes
            // logfiles that are no longer needed. Most applications
            // should use it.
            instance.Parameters.CircularLog = true;

            // Initialize the instance. This creates the logs and temporary database.
            // If logs are present in the log directory then recovery will run
            // automatically.
            instance.Init();

            // Create a disposable wrapper around a new JET_SESID. All database
            // access is done with a session (JET_SESID). Transactions and database
            // record visibility is per-session. Do not share sessions between
            // threads, instead create different sessions for different threads.
            using (var session = new Session(instance))
            {
                JET_DBID dbid;

                // The database only has to be attached once per instance, but each
                // session has to open the database. Redundant JetAttachDatabase calls
                // are safe to make though.
                // Here we use the fact that Instance, Session, and Table objects all have
                // implicit conversions to the underlying JET_* types. This allows the
                // disposable wrappers to be used with any APIs that expect the JET_*
                // structures.
                Api.JetAttachDatabase(session, DatabaseName, AttachDatabaseGrbit.None);
                Api.JetOpenDatabase(session, DatabaseName, null, out dbid, OpenDatabaseGrbit.None);

                // Create a disposable wrapper around the JET_TABLEID.
                // A JET_TABLEID acts as a database cursor, it can be used to:
                //  - Seek to a record
                //  - Retrieve columns from a record
                //  - Insert/Update/Delete a record
                //  - Move to the next/previous record
                using (var table = new Table(session, dbid, TableName, OpenTableGrbit.None))
                {
                    // Load the columnids from the table. This should be done each time the database is attached
                    // because an offline defrag (esentutl /d) can change the name => columnid mapping.
                    IDictionary<string, JET_COLUMNID> columnids = Api.GetColumnDictionary(session, table);
                    columnidSymbol = columnids["symbol"];
                    columnidName = columnids["name"];
                    columnidPrice = columnids["price"];
                    columnidShares = columnids["shares_owned"];

                    // Dump records by the primary index; this will be in stock symbol order.
                    Console.WriteLine("** Populating the table");
                    PopulateTable(session, table);
                    DumpByIndex(session, table, null);

                    // The shares index is sparse; it only contains records where the
                    // shares_owned column is non null.
                    Console.WriteLine("** Owned shares");
                    DumpByIndex(session, table, "shares");

                    // The noshares index is sparse; it only contains records where the
                    // shares_owned column is null.
                    Console.WriteLine("** Companies with no owned shares");
                    DumpByIndex(session, table, "noshares");

                    // Use the price index to find stocks sorted by price.
                    Console.WriteLine("** Sorted by price");
                    DumpByIndex(session, table, "price");

                    // Seek to, and update an existing record.
                    Console.WriteLine("** Updating owned shares");
                    BuyShares(session, table, "SBUX", 50);
                    BuyShares(session, table, "MSFT", 100);
                    DumpByIndex(session, table, "shares");

                    // Delete a record.
                    Console.WriteLine("** Deleting EBAY");
                    DeleteStock(session, table, "EBAY");
                    DumpByIndex(session, table, "name");

                    // Create an index range over a prefix.
                    Console.WriteLine("** Company names starting with 'app'");
                    DumpByNamePrefix(session, table, "app");

                    // Create an empty index range.
                    Console.WriteLine("** Company names starting with 'xyz'");
                    DumpByNamePrefix(session, table, "xyz");

                    // Create an index range over multiple records.
                    Console.WriteLine("** Company names starting with 'm'");
                    DumpByNamePrefix(session, table, "m");

                    // Move to the start of an index.
                    Console.WriteLine("** Lowest price");
                    Api.JetSetCurrentIndex(session, table, "price");
                    Api.JetMove(session, table, JET_Move.First, MoveGrbit.None);
                    PrintOneRow(session, table);
                    Console.WriteLine();

                    // Move to the end of an index.
                    Console.WriteLine("** Highest price");
                    Api.JetSetCurrentIndex(session, table, "price");
                    Api.JetMove(session, table, JET_Move.Last, MoveGrbit.None);
                    PrintOneRow(session, table);
                    Console.WriteLine();

                    // Create a index range between two values.
                    Console.WriteLine("** Prices fom $10-$20");
                    DumpPriceRange(session, table, 1000, 2000);

                    // Use intersect indexes to find records that meet multiple criteria.
                    Console.WriteLine("** Select by name and price");
                    IntersectIndexes(session, table, "Apple", "Pear", 1000, 2000);
                }
            }
        }
    }

    /// <summary>
    /// Dump all records from the index.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to dump.</param>
    /// <param name="index">The index to use.</param>
    private static void DumpByIndex(JET_SESID sesid, JET_TABLEID tableid, string index)
    {
        Api.JetSetCurrentIndex(sesid, tableid, index);
        PrintAllRecords(sesid, tableid);
        Console.WriteLine();
    }

    /// <summary>
    /// Populate the stock table with a set of symbols and prices.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to use.</param>
    private static void PopulateTable(JET_SESID sesid, JET_TABLEID tableid)
    {
        // Create a disposable wrapper around JetBeginTransaction and JetCommitTransaction.
        using (var transaction = new Transaction(sesid))
        {
            InsertOneStock(sesid, tableid, "SBUX", "Starbucks Corp.", 988, 0);
            InsertOneStock(sesid, tableid, "MSFT", "Microsoft Corp.", 1965, 200);
            InsertOneStock(sesid, tableid, "AAPL", "Apple Inc.", 9000, 0);
            InsertOneStock(sesid, tableid, "GOOG", "Google Inc.", 31017, 0);
            InsertOneStock(sesid, tableid, "M", "Macy's Inc.", 1062, 0);
            InsertOneStock(sesid, tableid, "MCD", "MacDonald's Corp.", 1062, 0);
            InsertOneStock(sesid, tableid, "GE", "General Electric Company", 1650, 0);
            InsertOneStock(sesid, tableid, "MMM", "3M Company", 5662, 100);
            InsertOneStock(sesid, tableid, "IBM", "International Business Machines Corp.", 8352, 150);
            InsertOneStock(sesid, tableid, "EBAY", "eBay Inc.", 1445, 0);

            // Commit the transaction at the end of the using block!
            // If transaction.Commit isn't called then the transaction will 
            // automatically rollback when disposed (throwing away
            // the records that were just inserted).
            transaction.Commit(CommitTransactionGrbit.None);
        }
    }

    /// <summary>
    /// Insert information about one stock into the table.
    /// </summary>
    /// <remarks>The session must already be in a transaction.</remarks>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to insert into.</param>
    /// <param name="symbol">The stock symbol.</param>
    /// <param name="name">The name of the company.</param>
    /// <param name="price">The current price.</param>
    /// <param name="sharesOwned">Number of shares owned.</param>
    private static void InsertOneStock(
        JET_SESID sesid,
        JET_TABLEID tableid,
        string symbol,
        string name,
        int price,
        int sharesOwned)
    {
        // Prepare an update, set some columns, and then save the update.
        // First create a disposable wrapper around JetPrepareUpdate and JetUpdate.
        using (var update = new Update(sesid, tableid, JET_prep.Insert))
        {
            Api.SetColumn(sesid, tableid, columnidSymbol, symbol, Encoding.Unicode);
            Api.SetColumn(sesid, tableid, columnidName, name, Encoding.Unicode);
            Api.SetColumn(sesid, tableid, columnidPrice, price);

            // This column defaults to null. Only set it if we have some shares.
            if (0 != sharesOwned)
            {
                Api.SetColumn(sesid, tableid, columnidShares, sharesOwned);
            }

            // Save the update at the end of the using block!
            // If update.Save isn't called then the update will 
            // be canceled when disposed (and the record won't
            // be inserted).
            //
            // Inserting a record does not change the location of
            // the cursor (JET_TABLEID); it will have the same
            // location that it did before the insert.
            // To insert a record and then position the cursor
            // on the record, use Update.SaveAndGotoBookmark. That
            // call uses the bookmark returned from JetUpdate to
            // position the tableid on the new record.
            update.Save();
        }
    }

    /// <summary>
    /// Increment the number of shares for a specified symbol.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to insert into.</param>
    /// <param name="symbol">The stock symbol to buy.</param>
    /// <param name="shares">The number of shares to buy.</param>
    private static void BuyShares(
        JET_SESID sesid,
        JET_TABLEID tableid,
        string symbol,
        int shares)
    {
        // Seek to the record, get the current value of the column,
        // prepare an update, set the column to the new value and the
        // save the update.
        using (var transaction = new Transaction(sesid))
        {
            SeekToSymbol(sesid, tableid, symbol);
            int currentShares = Api.RetrieveColumnAsInt32(sesid, tableid, columnidShares).GetValueOrDefault();
            int newShares = currentShares + shares;

            using (var update = new Update(sesid, tableid, JET_prep.Replace))
            {
                Api.SetColumn(sesid, tableid, columnidShares, newShares);

                // Save the update at the end of the using block!
                // If update.Save isn't called then the update will 
                // be cancelled when disposed (undoing the updates 
                // to the record).
                update.Save();
            }

            // Commit the transaction at the end of the using block!
            // If transaction.Commit isn't called then the transaction will 
            // automatically rollback when disposed (throwing away
            // the changes to the record).
            transaction.Commit(CommitTransactionGrbit.None);
        }
    }

    /// <summary>
    /// Delete the stock with the given symbol.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table cursor to use.</param>
    /// <param name="symbol">The symbol to delete.</param>
    private static void DeleteStock(JET_SESID sesid, JET_TABLEID tableid, string symbol)
    {
        // Seek to the record and delete it.
        using (var transaction = new Transaction(sesid))
        {
            SeekToSymbol(sesid, tableid, symbol);
            Api.JetDelete(sesid, tableid);

            // Commit the transaction at the end of the using block!
            // If transaction.Commit isn't called then the transaction will 
            // automatically rollback when disposed (undeleting the record).
            transaction.Commit(CommitTransactionGrbit.None);
        }
    }

    /// <summary>
    /// Find the record with the given stock symbol.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to dump the records from.</param>
    /// <param name="symbol">The symbol to seek for.</param>
    private static void SeekToSymbol(JET_SESID sesid, JET_TABLEID tableid, string symbol)
    {
        // We need to be on the primary index (which indexes the 'symbol' column).
        Api.JetSetCurrentIndex(sesid, tableid, null);
        Api.MakeKey(sesid, tableid, symbol, Encoding.Unicode, MakeKeyGrbit.NewKey);

        // This seek expects the record to be present. To test for a record
        // use TrySeek(), which won't throw an exception if the record isn't
        // found.
        Api.JetSeek(sesid, tableid, SeekGrbit.SeekEQ);
    }

    /// <summary>
    /// Dump all records where the name has the given prefix.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to dump.</param>
    /// <param name="namePrefix">The prefix to use.</param>
    private static void DumpByNamePrefix(JET_SESID sesid, JET_TABLEID tableid, string namePrefix)
    {
        // We are about to set up an index range on the name index.
        Api.JetSetCurrentIndex(sesid, tableid, "name");

        // First, seek to the beginning of the range.
        Api.MakeKey(sesid, tableid, namePrefix, Encoding.Unicode, MakeKeyGrbit.NewKey);
        if (Api.TrySeek(sesid, tableid, SeekGrbit.SeekGE))
        {
            // We are on the first record with name >= namePrefix. This record may not
            // be in the index range! We will now set the end of the index range, which
            // will tell us if the range is empty.
            Api.MakeKey(sesid, tableid, namePrefix, Encoding.Unicode, MakeKeyGrbit.NewKey | MakeKeyGrbit.SubStrLimit);
            if (Api.TrySetIndexRange(sesid, tableid, SetIndexRangeGrbit.RangeUpperLimit | SetIndexRangeGrbit.RangeInclusive))
            {
                // There are records in the range. We can now iterate through the range.
                // When the end of the range is hit we will get a 'no more records' error and
                // the range will be removed (so subsequent moves will go to the end of the table).
                PrintRecordsToEnd(sesid, tableid);
            }
        }

        Console.WriteLine();
    }

    /// <summary>
    /// Dump all records from the index whose price is in the given range.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to dump.</param>
    /// <param name="low">The low end of the price range.</param>
    /// <param name="high">The high end of the price range.</param>
    private static void DumpPriceRange(JET_SESID sesid, JET_TABLEID tableid, int low, int high)
    {
        // We are about to set up an index range on the price index.
        Api.JetSetCurrentIndex(sesid, tableid, "price");

        // First, seek to the beginning of the range.
        Api.MakeKey(sesid, tableid, low, MakeKeyGrbit.NewKey);
        if (Api.TrySeek(sesid, tableid, SeekGrbit.SeekGE))
        {
            Api.MakeKey(sesid, tableid, high, MakeKeyGrbit.NewKey);
            if (Api.TrySetIndexRange(sesid, tableid, SetIndexRangeGrbit.RangeUpperLimit))
            {
                // There are records in the range. We can now iterate through the range.
                // When the end of the range is hit we will get a 'no more records' error and
                // the range will be removed (so subsequent moves will go to the end of the table).
                PrintRecordsToEnd(sesid, tableid);
            }
        }

        Console.WriteLine();
    }

    /// <summary>
    /// Print records for companies whose name is between firstName and lastName and whose
    /// price is between lowPrice and highPrice. This is done by intersecting two different
    /// indexes, which can be an efficient way of finding records that meet multiple criteria.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to use.</param>
    /// <param name="fromName">The first name of the companies to dump.</param>
    /// <param name="toName">The last name of the companies to dump.</param>
    /// <param name="lowPrice">The lowest price to dump.</param>
    /// <param name="highPrice">The highest price to dump.</param>
    private static void IntersectIndexes(
        JET_SESID sesid,
        JET_TABLEID tableid,
        string fromName,
        string toName,
        int lowPrice,
        int highPrice)
    {
        // We need one cursor for each index. Remember to close these cursors
        // to avoid resource leaks. Here we use JetDupCursor, but JetOpenTable
        // would work as well.
        JET_TABLEID nameIndex;
        Api.JetDupCursor(sesid, tableid, out nameIndex, DupCursorGrbit.None);
        Api.JetSetCurrentIndex(sesid, nameIndex, "name");

        JET_TABLEID priceIndex;
        Api.JetDupCursor(sesid, tableid, out priceIndex, DupCursorGrbit.None);
        Api.JetSetCurrentIndex(sesid, priceIndex, "price");

        // Set the index range on the name index
        Api.MakeKey(sesid, nameIndex, fromName, Encoding.Unicode, MakeKeyGrbit.NewKey);
        Api.JetSeek(sesid, nameIndex, SeekGrbit.SeekLE);
        Api.MakeKey(sesid, nameIndex, toName, Encoding.Unicode, MakeKeyGrbit.NewKey);
        Api.JetSetIndexRange(sesid, nameIndex, SetIndexRangeGrbit.RangeInclusive | SetIndexRangeGrbit.RangeUpperLimit);

        // Set the index range on the prices
        Api.MakeKey(sesid, priceIndex, lowPrice, MakeKeyGrbit.NewKey);
        Api.JetSeek(sesid, priceIndex, SeekGrbit.SeekLE);
        Api.MakeKey(sesid, priceIndex, highPrice, MakeKeyGrbit.NewKey);
        Api.JetSetIndexRange(sesid, priceIndex, SetIndexRangeGrbit.RangeInclusive | SetIndexRangeGrbit.RangeUpperLimit);

        // This call will generate a set of bookmarks. The bookmarks are
        // returned in primary key order (for this table that will be symbol
        // order).
        foreach (byte[] bookmark in Api.IntersectIndexes(sesid, nameIndex, priceIndex))
        {
            Api.JetGotoBookmark(sesid, tableid, bookmark, bookmark.Length);
            PrintOneRow(sesid, tableid);
        }

        Console.WriteLine();

        Api.JetCloseTable(sesid, nameIndex);
        Api.JetCloseTable(sesid, priceIndex);
    }

    /// <summary>
    /// Print all rows in the table.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to dump the records from.</param>
    private static void PrintAllRecords(JET_SESID sesid, JET_TABLEID tableid)
    {
        if (Api.TryMoveFirst(sesid, tableid))
        {
            PrintRecordsToEnd(sesid, tableid);
        }
    }

    /// <summary>
    /// Print records from the current point to the end of the table.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to dump the records from.</param>
    private static void PrintRecordsToEnd(JET_SESID sesid, JET_TABLEID tableid)
    {
        do
        {
            PrintOneRow(sesid, tableid);
        }
        while (Api.TryMoveNext(sesid, tableid));
    }

    /// <summary>
    /// Print the current record.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">The table to use.</param>
    private static void PrintOneRow(JET_SESID sesid, JET_TABLEID tableid)
    {
        string symbol = Api.RetrieveColumnAsString(sesid, tableid, columnidSymbol);
        string name = Api.RetrieveColumnAsString(sesid, tableid, columnidName);

        // This column can't be null so we cast to an int.
        int price = (int)Api.RetrieveColumnAsInt32(sesid, tableid, columnidPrice);

        // This column can be null so we keep the nullable type.
        int? shares = Api.RetrieveColumnAsInt32(sesid, tableid, columnidShares);

        Console.Write("\t{0,-40} {1,-4} ${2,-6}", name, symbol, price / 100.0);
        if (shares.HasValue)
        {
            Console.Write(" {0} shares", shares);
        }

        Console.WriteLine();
    }

    /// <summary>
    /// Create a new database. Create the table, columns, and indexes.
    /// </summary>
    /// <param name="database">Name of the database to create.</param>
    private static void CreateDatabase(string database)
    {
        using (var instance = new Instance("createdatabase"))
        {
            instance.Init();
            using (var session = new Session(instance))
            {
                JET_DBID dbid;
                Api.JetCreateDatabase(session, database, null, out dbid, CreateDatabaseGrbit.OverwriteExisting);
                using (var transaction = new Transaction(session))
                {
                    // A newly created table is opened exclusively. This is necessary to add
                    // a primary index to the table (a primary index can only be added if the table
                    // is empty and opened exclusively). Columns and indexes can be added to a 
                    // table which is opened normally.
                    // The other way to create a table is to use JetCreateTableColumnIndex to
                    // add all columns and indexes with one call.
                    JET_TABLEID tableid;
                    Api.JetCreateTable(session, dbid, TableName, 16, 100, out tableid);
                    CreateColumnsAndIndexes(session, tableid);
                    Api.JetCloseTable(session, tableid);

                    // Lazily commit the transaction. Normally committing a transaction forces the
                    // associated log records to be flushed to disk, so the commit has to wait for
                    // the I/O to complete. Using the LazyFlush option means that the log records
                    // are kept in memory and will be flushed later. This will preserve transaction
                    // atomicity (all operations in the transaction will either happen or be rolled
                    // back) but will not preserve durability (a crash after the commit call may
                    // result in the transaction updates being lost). Lazy transaction commits are
                    // considerably faster though, as they don't have to wait for an I/O.
                    transaction.Commit(CommitTransactionGrbit.LazyFlush);
                }
            }
        }
    }

    /// <summary>
    /// Setup the meta-data for the given table.
    /// </summary>
    /// <param name="sesid">The session to use.</param>
    /// <param name="tableid">
    /// The table to add the columns/indexes to. This table must be opened exclusively.
    /// </param>
    private static void CreateColumnsAndIndexes(JET_SESID sesid, JET_TABLEID tableid)
    {
        using (var transaction = new Transaction(sesid))
        {
            JET_COLUMNID columnid;

            // Stock symbol : text column
            var columndef = new JET_COLUMNDEF
                                  {
                                      coltyp = JET_coltyp.LongText,
                                      cp = JET_CP.Unicode
                                  };

            Api.JetAddColumn(sesid, tableid, "symbol", columndef, null, 0, out columnid);

            // Name of the company : text column
            Api.JetAddColumn(sesid, tableid, "name", columndef, null, 0, out columnid);

            // Current price, stored in cents : 32-bit integer
            columndef = new JET_COLUMNDEF
            {
                coltyp = JET_coltyp.Long,

                // Be careful with ColumndefGrbit.ColumnNotNULL. Older versions of ESENT
                // (e.g. Windows XP) do not support this grbit for tagged or variable columns
                // (JET_coltyp.Text, JET_coltyp.LongText, JET_coltyp.Binary, JET_coltyp.LongBinary)
                grbit = ColumndefGrbit.ColumnNotNULL
            };

            Api.JetAddColumn(sesid, tableid, "price", columndef, null, 0, out columnid);

            // Number of shares owned (this column may be null) : 32-bit integer
            columndef.grbit = ColumndefGrbit.None;
            Api.JetAddColumn(sesid, tableid, "shares_owned", columndef, null, 0, out columnid);

            // Now add indexes. An index consists of several index segments (see
            // EsentVersion.Capabilities.ColumnsKeyMost to determine the maximum number of
            // segments). Each segment consists of a sort direction ('+' for ascending,
            // '-' for descending), a column name, and a '\0' separator. The index definition
            // must end with "\0\0". The count of characters should include all terminators.

            // The primary index is the stock symbol. The primary index is always unique.
            string indexDef = "+symbol\0\0";
            Api.JetCreateIndex(sesid, tableid, "primary", CreateIndexGrbit.IndexPrimary, indexDef, indexDef.Length, 100);

            // An index on the company name.
            indexDef = "+name\0+symbol\0\0";
            Api.JetCreateIndex(sesid, tableid, "name", CreateIndexGrbit.IndexUnique, indexDef, indexDef.Length, 100);

            // An index on the price. This index is not unique.
            indexDef = "+price\0\0";
            Api.JetCreateIndex(sesid, tableid, "price", CreateIndexGrbit.None, indexDef, indexDef.Length, 100);

            // Create 2 indexes that contain either companies where we own shares or companies
            // where we don't own shares. To do this we make the indexes conditional on the
            // 'shares_owned' column being null or non-null. When the 'shares_owned' column
            // is null entries will only appear in the 'noshares' index. When the 'shares_owned'
            // column is non-null (i.e. it has a value) entries will only appear in the 'shares'
            // index. Here we don't index the 'shares_owned' column (that would be valid too),
            // we just use it to determine membership in the index.
            const string IndexKey = "+name\0\0";
            JET_INDEXCREATE[] indexcreates = new[]
            {
                new JET_INDEXCREATE
                {
                    szIndexName = "shares",
                    szKey = IndexKey,
                    cbKey = IndexKey.Length,
                    rgconditionalcolumn = new[]
                    {
                        new JET_CONDITIONALCOLUMN
                        {
                            szColumnName = "shares_owned",
                            grbit = ConditionalColumnGrbit.ColumnMustBeNonNull
                        }
                    },
                    cConditionalColumn = 1
                },
            };

            // This is important: only create one index at a time with JetCreateIndex2!
            // The API takes an array of JET_INDEXCREATE objects, but if more than one
            // index is passed in then the API operates in batch mode, which requires
            // the caller NOT be in a transaction.
            Api.JetCreateIndex2(sesid, tableid, indexcreates, indexcreates.Length);

            // Now the first index has been created we change the name and invert the
            // condition and create a second index.
            indexcreates[0].szIndexName = "noshares";
            indexcreates[0].rgconditionalcolumn[0].grbit = ConditionalColumnGrbit.ColumnMustBeNull;
            Api.JetCreateIndex2(sesid, tableid, indexcreates, indexcreates.Length);

            transaction.Commit(CommitTransactionGrbit.LazyFlush);
        }
    }
}

Updated Wiki: SystemStats

laurionb — Wed, 16 Feb 2011 19:34:45 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

To collect performance counters we just add a new Sample to the dictionary. A PersistentDictionary is backed by an ESENT database which will always be consistent after a crash. That means we don't need to worry about data consistency after a crash.

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Updates to a PersistentDictionary are logged lazily -- after a crash some updates may be lost. It is possible to force the updates to be written to disk by calling the Flush() method on the PersistentDictionary, but that has to perform an I/O so it is considerably slower than a lazy update. In this case we are willing to lose a few updates after a crash.

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Querying the PersistentDictionary

The PersistentDictionary supports LINQ queries. Given a query like this:

DateTime startTime = ...;
DateTime endTime = ...;
IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

Only records which match the key criteria will be retrieved from the database. That means the runtime is determined by the number of samples retrieved, not the total number of samples. For best performance we only want to iterate over the samples once so we accumulate all the statistics at the same time, using the Aggregator class:

/// <summary>
/// Dump the aggregate stats for the specified time period.
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
/// <param name="startTime">The starting time.</param>
/// <param name="endTime">The ending time.</param>
private static void DumpStats(string dictionaryPath, DateTime startTime, DateTime endTime)
{
    if (!PersistentDictionaryFile.Exists(dictionaryPath))
    {
        Console.WriteLine("No stats collected");
        return;
    }

    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        Stopwatch queryTimer = Stopwatch.StartNew();
        IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

        Aggregator processor = new Aggregator();
        Aggregator paging = new Aggregator();
        Aggregator freeBytes = new Aggregator();
        foreach (Sample s in samples)
        {
            processor.AddSample(s.ProcessorTime);
            paging.AddSample(s.Paging);
            freeBytes.AddSample(s.FreeBytes);
        }

        queryTimer.Stop();
        Console.WriteLine("{0} samples from {1} to {2}", processor.NumSamples, startTime, endTime);
        Console.WriteLine("Processor time: {0}", processor);
        Console.WriteLine("Paging: {0}", paging);
        Console.WriteLine("Free bytes: {0}", freeBytes);
        Console.WriteLine("Query took {0}", queryTimer.Elapsed);
    }
}

The PersistentDictionary is multi-thread safe so it is fine to run (multiple) queries while the dictionary is being updated. It is possible to have multiple threads updating the dictionary as well.

Putting it all together

Now we just need a main method.

/// <summary>
/// The main method. Called on program startup.
/// </summary>
/// <param name="args">Arguments to the program.</param>
public static void Main(string[] args)
{
    const string DictionaryPath = "SystemStats";

    if (0 == args.Length)
    {
        CollectStats(DictionaryPath);
    }
    else if (2 == args.Length)
    {
        DumpStats(DictionaryPath, DateTime.Parse(args[0]), DateTime.Parse(args[1]));
    }
    else
    {
        if (!PersistentDictionaryFile.Exists(DictionaryPath))
        {
            Console.WriteLine("No stats collected");
        }
        else
        {
            using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(DictionaryPath))
            {
                // Getting the first item, last item or count of items are all O(1) operations.
                Console.WriteLine(
                    "Stats available from {0} to {1} ({2}) records",
                    dictionary.Keys.FirstOrDefault(),
                    dictionary.Keys.LastOrDefault(),
                    dictionary.Count);
            }                    
        }
    }
}

Performance

These numbers are from a database where samples were taken 100 times/second, there are slightly over 500,000 entries and the database is around 1.5Gb.

Stats available from 2/15/2011 12:59:48 PM to 2/16/2011 11:30:15 AM (7560591) records

Here are the results of a few queries. Each run is a new process so none of the database is cached. In a real-world scenario a process might keep the dictionary open so subsequent queries would be even faster:

>.\SystemStats.exe "2/15/2011 4:34:30 PM" "2/15/2011 4:34:33 PM"
235 samples from 2/15/2011 4:34:30 PM to 2/15/2011 4:34:33 PM
Processor time: min = 0.00, max = 100.00, average = 52.47
Paging: min = 0.00, max = 872.37, average = 4.11
Free bytes: min = 3,659,247,616.00, max = 3,678,650,368.00, average = 3,667,921,270.74
Query took 00:00:00.1053856

>.\SystemStats.exe "2/15/2011 2:13:00 PM" "2/15/2011 2:15:00 PM"
11805 samples from 2/15/2011 2:13:00 PM to 2/15/2011 2:15:00 PM
Processor time: min = 0.00, max = 100.00, average = 38.04
Paging: min = 0.00, max = 4,100.28, average = 3.39
Free bytes: min = 4,026,720,256.00, max = 4,069,093,376.00, average = 4,063,535,199.76
Query took 00:00:00.8090867

>.\SystemStats.exe "2/15/2011 5:23:00 PM" "2/15/2011 6:25:00 PM"
346751 samples from 2/15/2011 5:23:00 PM to 2/15/2011 6:25:00 PM
Processor time: min = 0.00, max = 100.00, average = 37.36
Paging: min = 0.00, max = 36,016.09, average = 11.27
Free bytes: min = 3,183,525,888.00, max = 3,589,419,008.00, average = 3,413,498,109.31
Query took 00:00:21.6863322

The query performance is proportional to the number of entries processed, it will be basically independent of the database size.

Updated Wiki: SystemStats

martinc — Wed, 16 Feb 2011 01:04:04 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Querying the PersistentDictionary

The PersistentDictionary supports LINQ queries. Given a query like this:

DateTime startTime = ...;
DateTime endTime = ...;
IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

/// <summary>
/// Dump the aggregate stats for the specified time period.
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
/// <param name="startTime">The starting time.</param>
/// <param name="endTime">The ending time.</param>
private static void DumpStats(string dictionaryPath, DateTime startTime, DateTime endTime)
{
    if (!PersistentDictionaryFile.Exists(dictionaryPath))
    {
        Console.WriteLine("No stats collected");
        return;
    }

    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        Stopwatch queryTimer = Stopwatch.StartNew();
        IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

        Aggregator processor = new Aggregator();
        Aggregator paging = new Aggregator();
        Aggregator freeBytes = new Aggregator();
        foreach (Sample s in samples)
        {
            processor.AddSample(s.ProcessorTime);
            paging.AddSample(s.Paging);
            freeBytes.AddSample(s.FreeBytes);
        }

        queryTimer.Stop();
        Console.WriteLine("{0} samples from {1} to {2}", processor.NumSamples, startTime, endTime);
        Console.WriteLine("Processor time: {0}", processor);
        Console.WriteLine("Paging: {0}", paging);
        Console.WriteLine("Free bytes: {0}", freeBytes);
        Console.WriteLine("Query took {0}", queryTimer.Elapsed);
    }
}

The PersistentDictionary is multi-thread safe so it is fine to run (multiple) queries while the dictionary is being updated. It is possible to have multiple threads updating the dictionary as well.

Putting it all together

Now we just need a main method.

/// <summary>
/// The main method. Called on program startup.
/// </summary>
/// <param name="args">Arguments to the program.</param>
public static void Main(string[] args)
{
    const string DictionaryPath = "SystemStats";

    if (0 == args.Length)
    {
        CollectStats(DictionaryPath);
    }
    else if (2 == args.Length)
    {
        DumpStats(DictionaryPath, DateTime.Parse(args[0]), DateTime.Parse(args[1]));
    }
    else
    {
        if (!PersistentDictionaryFile.Exists(DictionaryPath))
        {
            Console.WriteLine("No stats collected");
        }
        else
        {
            using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(DictionaryPath))
            {
                // Getting the first item, last item or count of items are all O(1) operations.
                Console.WriteLine(
                    "Stats available from {0} to {1} ({2}) records",
                    dictionary.Keys.FirstOrDefault(),
                    dictionary.Keys.LastOrDefault(),
                    dictionary.Count);
            }                    
        }
    }
}

Performance

These numbers are from a database where samples were taken 100 times/second, there are slightly over 500,000 entries and the database is around 110MB.

Stats available from 2/15/2011 12:59:48 PM to 2/15/2011 2:33:19 PM (492174) records

Here are the results of a few queries. Each run is a new process so none of the database is cached. In a real-world scenario a process might keep the dictionary open so subsequent queries would be even faster:

>.\SystemStats.exe "2/15/2011 2:43:00 PM" "2/15/2011 2:43:01 PM"
82 samples from 2/15/2011 2:43:00 PM to 2/15/2011 2:43:01 PM
Processor time: min = 0.00, max = 100.00, average = 42.89
Paging: min = 0.00, max = 464.90, average = 6.28
Free bytes: min = 3,873,783,808.00, max = 3,911,086,080.00, average = 3,901,525,966.05
Query took 00:00:00.1149587

>.\SystemStats.exe "2/15/2011 2:33:00 PM" "2/15/2011 2:34:00 PM"
1795 samples from 2/15/2011 2:33:00 PM to 2/15/2011 2:34:00 PM
Processor time: min = 0.00, max = 100.00, average = 41.26
Paging: min = 0.00, max = 3,075.65, average = 7.93
Free bytes: min = 3,995,095,040.00, max = 4,011,421,696.00, average = 4,005,309,211.24
Query took 00:00:00.2190249

>.\SystemStats.exe "2/15/2011 2:34:30 PM" "2/15/2011 2:49:30 PM"
64002 samples from 2/15/2011 2:34:30 PM to 2/15/2011 2:49:30 PM
Processor time: min = 0.00, max = 100.00, average = 40.30
Paging: min = 0.00, max = 11,241.36, average = 4.79
Free bytes: min = 3,860,332,544.00, max = 4,047,314,944.00, average = 3,927,072,817.73
Query took 00:00:03.9911891

>.\SystemStats.exe "2/15/2011 1:27:48 PM" "2/15/2011 2:27:48 PM"
337585 samples from 2/15/2011 1:27:48 PM to 2/15/2011 2:27:48 PM
Processor time: min = 0.00, max = 100.00, average = 37.83
Paging: min = 0.00, max = 54,169.92, average = 68.10
Free bytes: min = 3,986,374,656.00, max = 4,128,804,864.00, average = 4,068,317,649.07
Query took 00:00:20.6299002

The query performance is proportional to the number of entries processed, it will be basically independent of the database size.

Updated Wiki: PersistentDictionaryDocumentation

laurionb — Tue, 15 Feb 2011 23:29:10 GMT

Data Storage with PersistentDictionary

PersistentDictionary is a database-backed dictionary built on top of the ESENT database engine. It is a drop-in compatible replacement for the generic Dictionary<TKey,TValue>, SortedDictionary<TKey, TValue> and SortedList<TKey, TValue> classes found in the System.Collections.Generic namespace.

class PersistentDictionary<TKey, TValue> : IDictionary<TKey, TValue>, ICollection<KeyValuePair<TKey, TValue>>, IEnumerable<KeyValuePair<TKey, TValue>>, IDisposable
    where TKey : IComparable<TKey>

PersistentDictionary Features

No setup: the ESENT database engine is part of Windows and requires no setup. EsentCollections will work on any version of Windows from XP onwards.
Performance: ESENT supports a high rate of updates and retrieves. Write-ahead logging decreases the cost of making small updates to the data. Information is inserted into or retrieved from the database in-process so data access has very low overhead. B-trees give O(log n) access to data by key and the records are stored in sorted order.
Simplicity: a PersistentDictionary looks and behaves like the .NET Dictionary/SortedDictionary/SortedList classes. No extra method calls are required.
Administration-free: ESENT automatically manages the database cache size, transaction logfiles, and crash recovery so no database administration is needed. The code is structured so that there are no deadlocks or conflicts, even when multiple threads use the same dictionary. ESENT runs in-process and doesn’t expose any network access, providing a high degree of security.
Reliability: ESENT’s write-ahead logging system means that a database is automatically recovered after a process crash or unexpected machine shutdown (e.g. power outage). Database transactions are used to ensure the logical consistency of the database.
Concurrency: each data structure can be accessed by multiple threads. Reads are non-blocking and updates to different items in the collection are allowed to proceed concurrently.
Scale: A collection can contain up to 2^31 objects, and values can be up to 2GB in size. The maximum database size is 16TB .

Sample Code

Here is an application that remembers a first name → last name mapping in a persistent dictionary.

using System;
using Microsoft.Isam.Esent.Collections.Generic;

namespace PersistentDictionarySample
{
    public static class HelloWorld
    {
        public static void Main()
        {
            var dictionary = new PersistentDictionary<string, string>("Names");
            Console.WriteLine("What is your first name?");
            string firstName = Console.ReadLine();
            if (dictionary.ContainsKey(firstName))
            {
                Console.WriteLine("Welcome back {0} {1}",
                    firstName,
                    dictionary[firstName]);
            }
            else
            {
                Console.WriteLine(
                    "I don't know you, {0}. What is your last name?",
                    firstName);
                dictionary[firstName] = Console.ReadLine();
            }
        }
    }
}

Sample Code in other languages

VB.NET Sample
C++/CLI Sample

Sample Application

A more sophisticated sample application that tracks some perf counters and performs a simple LINQ query.

Quick Start

To get started with PersistentDictionary you should:

Download the latest version of the PersistentDictionary project. http://managedesent.codeplex.com/releases.
Copy Esent.Interop.dll and Esent.Collections.dll to your project.
Add a reference to Esent.Collections.dll.
Use the namespace ‘Microsoft.Isam.Esent.Collections.Generic’.
Pick a directory to contain the persistent dictionary. The directory will contain all the files (database, logfiles, and checkpoint) needed for the database that stores the data.
Create a PersistentDictionary, passing the directory name into the constructor.
Use the PersistentDictionary like an ordinary Dictionary or SortedDictionary.
Dispose of the PersistentDictionary when finished with it. If the dictionary isn’t disposed then ESENT will have the database open until the instance is finalized.

Supported Key Types

Only these types are supported as dictionary keys:

Boolean	Byte	Int16
UInt16	Int32	UInt32
Int64	UInt64	Float
Double	Guid	DateTime
TimeSpan	String

Supported Value Types

Dictionary values can be any of the key types, Nullable<T> versions of the key types, Uri, IPAddress or a serializable structure. A structure is only considered serializable if it meets all these criteria:

The structure is marked as serializable
Every member of the struct is either:
- A primitive data type (e.g. Int32)
- A String, Uri or IPAddress
- A serializable structure.

Or, to put it another way, a serializable structure cannot contain any references to a class object. This is done to preserve API consistency. Adding an object to a PersistentDictionary creates a copy of the object though serialization. Modifying the original object will not modify the copy, which would lead to confusing behavior. To avoid those problems the PersistentDictionary will only accept value types as values.

Can Be Serialized

[Serializable]
struct Good
{
    public DateTime? Received;
    public string Name;
    public Decimal Price;
    public Uri Url;
}

Can’t Be Serialized

[Serializable]
struct Bad
{
    public byte[] Data; // arrays aren’t supported
    public Exception Error; // reference object 
}

PersistentDictionary API

PersistentDictionary implements the generic IDictionary interface. See the MSDN entry for System.Collections.Generic.IDictionary for documentation. Additional methods are:

Constructors
The constructor has to specify the location of the database files.
PersistentDictionary(string directory) : Create a PersistentDictionary in the specified directory.
PersistentDictionary(IEnumerable<KeyValuePair<TKey, TValue>> dictionary, string directory) : Create a PersistentDictionary in the specified directory, copying entries from the specified collection.

Methods
PersistentDictionary.Flush() : Force all changes made to this dictionary to be written to disk.

File Manipulation
These are static methods of the PersistentDictionaryFile class.
PersistentDictionaryFile.Exists(string directory) : Determine if a dictionary database file exists in the specified directory.
PersistentDictionaryFile.DeleteFiles(string directory) : Delete all files associated with a PersistedDictionary database from the specified directory.

LINQ Support

A PersistentDictionary can optimize some LINQ operators by retrieving only matching records from the database. For this to happen the LINQ query should specify a subset of item keys using comparison operators. Supported operators are: <, <=, ==, !=, >, >=, Equals, CompareTo and StartsWith (for strings). LINQ queries that only examine values cannot be optimized at all.

Besides 'where', these LINQ operators can be optimized: Any, Min, Max, First, FirstOrDefault, Last, LastOrDefault, Single, SingleOrDefault, Count, Reverse.

Examples of LINQ statements which have efficient support:

var q = persistentDictionary.Where(x => x.Key < 5 && x.Key > 2 && x.Key != 4).Reverse();
var q = from x in persistentDictionary where x.Key.StartsWith("de") || x.Key.StartsWith("bi") select x.Value;
if (persistentDictionary.Keys.Any(x => x < 5)) { ... }
var q = from x in persistentDictionary where (x.Key.CompareTo("a") > 0 && x.Key.CompareTo("c") <= 0) select x.Value;

Data Consistency

Each database update is performed in a separate transaction and the logfile and database updates are performed in the background. This means that dictionary updates are atomic and consistent, but not always durable. If the application crashes only updates whose log records have been written to disk will be recovered. It is possible to force the dictionary updates created so far to be persisted to disk using PersistentDictionary.Flush(). Note that every Flush() call requires a disk I/O so using this too frequently will severely limit the update rate. Disposing the dictionary also flushes all its changes to disk.

Performance Measurements

These are very basic performance measurements made with a PersistentDictionary<long, string> where the string data was 64 bytes in length. These measurements were taken on my desktop system.

Sequential inserts	32,000 entries/second
Random inserts	17,000 entries/second
Random Updates	36,000 entries/second
Random lookups (database cached in memory)	137,000 entries/second
Linq queries (range of records)	14,000 queries/second

Performance will vary from application to application. Factors that affect performance include:

Total number of items in the dictionary.
Size of the data. Retrieving an int will be faster than retrieving a 55MB string.
How much of the database is cached in memory. When first opening a populated dictionary the data will not be cached in memory so initial lookups will be slow.
Update patterns. Sequential inserts are much faster than random inserts.
Structure serialization. Using structures as dictionary data can be much slower than using basic data types.
Disk performance.

File Management

The PersistentDictionary code will automatically open or create a database in the specified directory. To determine if a dictionary database already exists use PersistentDictionaryFile.Exists(). To remove a dictionary use PersistentDictionaryFile.DeleteFiles().

Updated Wiki: PersistentDictionaryDocumentation

laurionb — Tue, 15 Feb 2011 22:56:34 GMT

Data Storage with PersistentDictionary

class PersistentDictionary<TKey, TValue> : IDictionary<TKey, TValue>, ICollection<KeyValuePair<TKey, TValue>>, IEnumerable<KeyValuePair<TKey, TValue>>, IDisposable
    where TKey : IComparable<TKey>

PersistentDictionary Features

No setup: the ESENT database engine is part of Windows and requires no setup. EsentCollections will work on any version of Windows from XP onwards.
Performance: ESENT supports a high rate of updates and retrieves. Write-ahead logging decreases the cost of making small updates to the data. Information is inserted into or retrieved from the database in-process so data access has very low overhead. B-trees give O(log n) access to data by key and the records are stored in sorted order.
Simplicity: a PersistentDictionary looks and behaves like the .NET Dictionary/SortedDictionary/SortedList classes. No extra method calls are required.
Administration-free: ESENT automatically manages the database cache size, transaction logfiles, and crash recovery so no database administration is needed. The code is structured so that there are no deadlocks or conflicts, even when multiple threads use the same dictionary. ESENT runs in-process and doesn’t expose any network access, providing a high degree of security.
Reliability: ESENT’s write-ahead logging system means that a database is automatically recovered after a process crash or unexpected machine shutdown (e.g. power outage). Database transactions are used to ensure the logical consistency of the database.
Concurrency: each data structure can be accessed by multiple threads. Reads are non-blocking and updates to different items in the collection are allowed to proceed concurrently.
Scale: A collection can contain up to 2^31 objects, and values can be up to 2GB in size. The maximum database size is 16TB .

Sample Code

Here is an application that remembers a first name → last name mapping in a persistent dictionary.

using System;
using Microsoft.Isam.Esent.Collections.Generic;

namespace PersistentDictionarySample
{
    public static class HelloWorld
    {
        public static void Main()
        {
            var dictionary = new PersistentDictionary<string, string>("Names");
            Console.WriteLine("What is your first name?");
            string firstName = Console.ReadLine();
            if (dictionary.ContainsKey(firstName))
            {
                Console.WriteLine("Welcome back {0} {1}",
                    firstName,
                    dictionary[firstName]);
            }
            else
            {
                Console.WriteLine(
                    "I don't know you, {0}. What is your last name?",
                    firstName);
                dictionary[firstName] = Console.ReadLine();
            }
        }
    }
}

Sample Code in other languages

VB.NET Sample
C++/CLI Sample

Sample Application

A more sophisticated sample application that tracks some perf counters and performs a simple LINQ query.

Quick Start

To get started with PersistentDictionary you should:

Download the latest version of the PersistentDictionary project. http://managedesent.codeplex.com/releases.
Copy Esent.Interop.dll and Esent.Collections.dll to your project.
Add a reference to Esent.Collections.dll.
Use the namespace ‘Microsoft.Isam.Esent.Collections.Generic’.
Pick a directory to contain the persistent dictionary. The directory will contain all the files (database, logfiles, and checkpoint) needed for the database that stores the data.
Create a PersistentDictionary, passing the directory name into the constructor.
Use the PersistentDictionary like an ordinary Dictionary or SortedDictionary.
Dispose of the PersistentDictionary when finished with it. If the dictionary isn’t disposed then ESENT will have the database open until the instance is finalized.

Supported Key Types

Only these types are supported as dictionary keys:

Boolean	Byte	Int16
UInt16	Int32	UInt32
Int64	UInt64	Float
Double	Guid	DateTime
TimeSpan	String

Supported Value Types

The structure is marked as serializable
Every member of the struct is either:
- A primitive data type (e.g. Int32)
- A String, Uri or IPAddress
- A serializable structure.

[Serializable]
struct Good
{
    public DateTime? Received;
    public string Name;
    public Decimal Price;
    public Uri Url;
}

Can’t Be Serialized

[Serializable]
struct Bad
{
    public byte[] Data; // arrays aren’t supported
    public Exception Error; // reference object 
}

PersistentDictionary API

Data Consistency

Performance Measurements

These are very basic performance measurements made with a PersistentDictionary<long, string> where the string data was 64 bytes in length. These measurements were taken on my desktop system.

Sequential inserts	32,000 entries/second
Random inserts	17,000 entries/second
Random Updates	36,000 entries/second
Random lookups (database cached in memory)	137,000 entries/second
Linq queries (range of records)	14,000 queries/second

Performance will vary from application to application. Factors that affect performance include:

Total number of items in the dictionary.
Size of the data. Retrieving an int will be faster than retrieving a 55MB string.
How much of the database is cached in memory. When first opening a populated dictionary the data will not be cached in memory so initial lookups will be slow.
Update patterns. Sequential inserts are much faster than random inserts.
Structure serialization. Using structures as dictionary data can be much slower than using basic data types.
Disk performance.

File Management

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 22:54:29 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Querying the PersistentDictionary

The PersistentDictionary supports LINQ queries. Given a query like this:

DateTime startTime = ...;
DateTime endTime = ...;
IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

/// <summary>
/// Dump the aggregate stats for the specified time period.
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
/// <param name="startTime">The starting time.</param>
/// <param name="endTime">The ending time.</param>
private static void DumpStats(string dictionaryPath, DateTime startTime, DateTime endTime)
{
    if (!PersistentDictionaryFile.Exists(dictionaryPath))
    {
        Console.WriteLine("No stats collected");
        return;
    }

    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        Stopwatch queryTimer = Stopwatch.StartNew();
        IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

        Aggregator processor = new Aggregator();
        Aggregator paging = new Aggregator();
        Aggregator freeBytes = new Aggregator();
        foreach (Sample s in samples)
        {
            processor.AddSample(s.ProcessorTime);
            paging.AddSample(s.Paging);
            freeBytes.AddSample(s.FreeBytes);
        }

        queryTimer.Stop();
        Console.WriteLine("{0} samples from {1} to {2}", processor.NumSamples, startTime, endTime);
        Console.WriteLine("Processor time: {0}", processor);
        Console.WriteLine("Paging: {0}", paging);
        Console.WriteLine("Free bytes: {0}", freeBytes);
        Console.WriteLine("Query took {0}", queryTimer.Elapsed);
    }
}

The PersistentDictionary is multi-thread safe so it is fine to run (multiple) queries while the dictionary is being updated. It is possible to have multiple threads updating the dictionary as well.

Putting it all together

Now we just need a main method.

/// <summary>
/// The main method. Called on program startup.
/// </summary>
/// <param name="args">Arguments to the program.</param>
public static void Main(string[] args)
{
    const string DictionaryPath = "SystemStats";

    if (0 == args.Length)
    {
        CollectStats(DictionaryPath);
    }
    else if (2 == args.Length)
    {
        DumpStats(DictionaryPath, DateTime.Parse(args[0]), DateTime.Parse(args[1]));
    }
    else
    {
        if (!PersistentDictionaryFile.Exists(DictionaryPath))
        {
            Console.WriteLine("No stats collected");
        }
        else
        {
            using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(DictionaryPath))
            {
                // Getting the first item, last item or count of items are all O(1) operations.
                Console.WriteLine(
                    "Stats available from {0} to {1} ({2}) records",
                    dictionary.Keys.FirstOrDefault(),
                    dictionary.Keys.LastOrDefault(),
                    dictionary.Count);
            }                    
        }
    }
}

Performance

>.\SystemStats.exe "2/15/2011 2:43:00 PM" "2/15/2011 2:43:01 PM"
82 samples from 2/15/2011 2:43:00 PM to 2/15/2011 2:43:01 PM
Processor time: min = 0.00, max = 100.00, average = 42.89
Paging: min = 0.00, max = 464.90, average = 6.28
Free bytes: min = 3,873,783,808.00, max = 3,911,086,080.00, average = 3,901,525,966.05
Query took 00:00:00.1149587

>.\SystemStats.exe "2/15/2011 2:33:00 PM" "2/15/2011 2:34:00 PM"
1795 samples from 2/15/2011 2:33:00 PM to 2/15/2011 2:34:00 PM
Processor time: min = 0.00, max = 100.00, average = 41.26
Paging: min = 0.00, max = 3,075.65, average = 7.93
Free bytes: min = 3,995,095,040.00, max = 4,011,421,696.00, average = 4,005,309,211.24
Query took 00:00:00.2190249

>.\SystemStats.exe "2/15/2011 2:34:30 PM" "2/15/2011 2:49:30 PM"
64002 samples from 2/15/2011 2:34:30 PM to 2/15/2011 2:49:30 PM
Processor time: min = 0.00, max = 100.00, average = 40.30
Paging: min = 0.00, max = 11,241.36, average = 4.79
Free bytes: min = 3,860,332,544.00, max = 4,047,314,944.00, average = 3,927,072,817.73
Query took 00:00:03.9911891

>.\SystemStats.exe "2/15/2011 1:27:48 PM" "2/15/2011 2:27:48 PM"
337585 samples from 2/15/2011 1:27:48 PM to 2/15/2011 2:27:48 PM
Processor time: min = 0.00, max = 100.00, average = 37.83
Paging: min = 0.00, max = 54,169.92, average = 68.10
Free bytes: min = 3,986,374,656.00, max = 4,128,804,864.00, average = 4,068,317,649.07
Query took 00:00:20.6299002

The query performance is proportional to the number of entries processed, it will be basically independant of the database size.

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 21:52:37 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Querying the PersistentDictionary

The PersistentDictionary supports LINQ queries. Given a query like this:

DateTime startTime = ...;
DateTime endTime = ...;
IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

/// <summary>
/// Dump the aggregate stats for the specified time period.
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
/// <param name="startTime">The starting time.</param>
/// <param name="endTime">The ending time.</param>
private static void DumpStats(string dictionaryPath, DateTime startTime, DateTime endTime)
{
    if (!PersistentDictionaryFile.Exists(dictionaryPath))
    {
        Console.WriteLine("No stats collected");
        return;
    }

    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        Stopwatch queryTimer = Stopwatch.StartNew();
        IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

        Aggregator processor = new Aggregator();
        Aggregator paging = new Aggregator();
        Aggregator freeBytes = new Aggregator();
        foreach (Sample s in samples)
        {
            processor.AddSample(s.ProcessorTime);
            paging.AddSample(s.Paging);
            freeBytes.AddSample(s.FreeBytes);
        }

        queryTimer.Stop();
        Console.WriteLine("{0} samples from {1} to {2}", processor.NumSamples, startTime, endTime);
        Console.WriteLine("Processor time: {0}", processor);
        Console.WriteLine("Paging: {0}", paging);
        Console.WriteLine("Free bytes: {0}", freeBytes);
        Console.WriteLine("Query took {0}", queryTimer.Elapsed);
    }
}

The PersistentDictionary is multi-thread safe so it is fine to run (multiple) queries while the dictionary is being updated. It is possible to have multiple threads updating the dictionary as well.

Putting it all together

Now we just need a main method.

/// <summary>
/// The main method. Called on program startup.
/// </summary>
/// <param name="args">Arguments to the program.</param>
public static void Main(string[] args)
{
    const string DictionaryPath = "SystemStats";

    if (0 == args.Length)
    {
        CollectStats(DictionaryPath);
    }
    else if (2 == args.Length)
    {
        DumpStats(DictionaryPath, DateTime.Parse(args[0]), DateTime.Parse(args[1]));
    }
    else
    {
        if (!PersistentDictionaryFile.Exists(DictionaryPath))
        {
            Console.WriteLine("No stats collected");
        }
        else
        {
            using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(DictionaryPath))
            {
                // Getting the first item, last item or count of items are all O(1) operations.
                Console.WriteLine(
                    "Stats available from {0} to {1} ({2}) records",
                    dictionary.Keys.FirstOrDefault(),
                    dictionary.Keys.LastOrDefault(),
                    dictionary.Count);
            }                    
        }
    }
}

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 21:47:33 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Querying the PersistentDictionary

The PersistentDictionary supports LINQ queries. Given a query like this:

DateTime startTime = ...;
DateTime endTime = ...;
IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

/// <summary>
/// Dump the aggregate stats for the specified time period.
/// </summary>
/// <param name="dictionaryPath">The path to the dictionary.</param>
/// <param name="startTime">The starting time.</param>
/// <param name="endTime">The ending time.</param>
private static void DumpStats(string dictionaryPath, DateTime startTime, DateTime endTime)
{
    if (!PersistentDictionaryFile.Exists(dictionaryPath))
    {
        Console.WriteLine("No stats collected");
        return;
    }

    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        Stopwatch queryTimer = Stopwatch.StartNew();
        IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

        Aggregator processor = new Aggregator();
        Aggregator paging = new Aggregator();
        Aggregator freeBytes = new Aggregator();
        foreach (Sample s in samples)
        {
            processor.AddSample(s.ProcessorTime);
            paging.AddSample(s.Paging);
            freeBytes.AddSample(s.FreeBytes);
        }

        queryTimer.Stop();
        Console.WriteLine("{0} samples from {1} to {2}", processor.NumSamples, startTime, endTime);
        Console.WriteLine("Processor time: {0}", processor);
        Console.WriteLine("Paging: {0}", paging);
        Console.WriteLine("Free bytes: {0}", freeBytes);
        Console.WriteLine("Query took {0}", queryTimer.Elapsed);
    }
}

Putting it all together

Now we just need a main method.

/// <summary>
/// The main method. Called on program startup.
/// </summary>
/// <param name="args">Arguments to the program.</param>
public static void Main(string[] args)
{
    const string DictionaryPath = "SystemStats";

    if (0 == args.Length)
    {
        CollectStats(DictionaryPath);
    }
    else if (2 == args.Length)
    {
        DumpStats(DictionaryPath, DateTime.Parse(args[0]), DateTime.Parse(args[1]));
    }
    else
    {
        if (!PersistentDictionaryFile.Exists(DictionaryPath))
        {
            Console.WriteLine("No stats collected");
        }
        else
        {
            using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(DictionaryPath))
            {
                // Getting the first item, last item or count of items are all O(1) operations.
                Console.WriteLine(
                    "Stats available from {0} to {1} ({2}) records",
                    dictionary.Keys.FirstOrDefault(),
                    dictionary.Keys.LastOrDefault(),
                    dictionary.Count);
            }                    
        }
    }
}

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 21:36:06 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">
/// The path to the dictionary.
/// </param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Querying the PersistentDictionary

The PersistentDictionary supports LINQ queries. Given a query like this:

DateTime startTime = ...;
DateTime endTime = ...;
IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

/// <summary>
/// Dump the aggregate stats for the specified time period.
/// </summary>
/// <param name="startTime">The starting time.</param>
/// <param name="endTime">The ending time.</param>
private static void DumpStats(string dictionaryPath, DateTime startTime, DateTime endTime)
{
    if (!PersistentDictionaryFile.Exists(dictionaryPath))
    {
        Console.WriteLine("No stats collected");
        return;
    }

    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        Stopwatch queryTimer = Stopwatch.StartNew();
        IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

        Aggregator processor = new Aggregator();
        Aggregator paging = new Aggregator();
        Aggregator freeBytes = new Aggregator();
        foreach (Sample s in samples)
        {
            processor.AddSample(s.ProcessorTime);
            paging.AddSample(s.Paging);
            freeBytes.AddSample(s.FreeBytes);
        }

        queryTimer.Stop();
        Console.WriteLine("{0} samples from {1} to {2}", processor.NumSamples, startTime, endTime);
        Console.WriteLine("Processor time: {0}", processor);
        Console.WriteLine("Paging: {0}", paging);
        Console.WriteLine("Free bytes: {0}", freeBytes);
        Console.WriteLine("Query took {0}", queryTimer.Elapsed);
    }
}

Putting it all together

Now we just need a main method.

/// <summary>
/// The main method. Called on program startup.
/// </summary>
/// <param name="args">Arguments to the program.</param>
public static void Main(string[] args)
{
    const string DictionaryPath = "SystemStats";

    if (0 == args.Length)
    {
        CollectStats(DictionaryPath);
    }
    else if (2 == args.Length)
    {
        DumpStats(DictionaryPath, DateTime.Parse(args[0]), DateTime.Parse(args[1]));
    }
    else
    {
        if (!PersistentDictionaryFile.Exists(DictionaryPath))
        {
            Console.WriteLine("No stats collected");
        }
        else
        {
            using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(DictionaryPath))
            {
                Console.WriteLine(
                    "Stats available from {0} to {1} ({2}) records",
                    dictionary.Keys.FirstOrDefault(),
                    dictionary.Keys.LastOrDefault(),
                    dictionary.Count);
            }                    
        }
    }
}

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 21:34:42 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">
/// The path to the dictionary.
/// </param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Querying the PersistentDictionary

The PersistentDictionary supports LINQ queries. Given a query like this:

DateTime startTime = ...;
DateTime endTime = ...;
IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

/// <summary>
/// Dump the aggregate stats for the specified time period.
/// </summary>
/// <param name="startTime">The starting time.</param>
/// <param name="endTime">The ending time.</param>
private static void DumpStats(string dictionaryPath, DateTime startTime, DateTime endTime)
{
    if (!PersistentDictionaryFile.Exists(dictionaryPath))
    {
        Console.WriteLine("No stats collected");
        return;
    }

    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        Stopwatch queryTimer = Stopwatch.StartNew();
        IEnumerable<Sample> samples = from x in dictionary where x.Key >= startTime && x.Key <= endTime select x.Value;

        Aggregator processor = new Aggregator();
        Aggregator paging = new Aggregator();
        Aggregator freeBytes = new Aggregator();
        foreach (Sample s in samples)
        {
            processor.AddSample(s.ProcessorTime);
            paging.AddSample(s.Paging);
            freeBytes.AddSample(s.FreeBytes);
        }

        queryTimer.Stop();
        Console.WriteLine("{0} samples from {1} to {2}", processor.NumSamples, startTime, endTime);
        Console.WriteLine("Processor time: {0}", processor);
        Console.WriteLine("Paging: {0}", paging);
        Console.WriteLine("Free bytes: {0}", freeBytes);
        Console.WriteLine("Query took {0}", queryTimer.Elapsed);
    }
}

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 21:29:31 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">
/// The path to the dictionary.
/// </param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Calculating statistics

While querying the data we will want to calculate the min, max and average of the samples. This class will be used to do the calculations:

/// <summary>
/// Calculates the average, min and max of a set of samples.
/// </summary>
private class Aggregator
{
    /// <summary>
    /// Number of samples seen.
    /// </summary>
    private int numSamples = 0;

    /// <summary>
    /// Running total of all the samples.
    /// </summary>
    private double total = 0;

    /// <summary>
    /// The smallest sample seen.
    /// </summary>
    private double min = Double.MaxValue;

    /// <summary>
    /// The largest sample seen.
    /// </summary>
    private double max = Double.MinValue;

    /// <summary>
    /// Gets the number of samples that were seen.
    /// </summary>
    public int NumSamples
    {
        get { return this.numSamples; }
    }

    /// <summary>
    /// Gets the average of all the samples.
    /// </summary>
    private double Average
    {
        get { return 0 == this.numSamples ? 0 : this.total / this.numSamples; }
    }

    /// <summary>
    /// Add a new sample.
    /// </summary>
    /// <param name="sample">The sample value/</param>
    public void AddSample(double sample)
    {
        this.numSamples++;
        this.total += sample;
        this.min = Math.Min(this.min, sample);
        this.max = Math.Max(this.max, sample);
    }

    /// <summary>
    /// Gets a string representation of the aggregate calculation.
    /// </summary>
    /// <returns>
    /// A string representation of the aggregate calculation
    /// </returns>
    public override string ToString()
    {
        return String.Format("min = {0:N2}, max = {1:N2}, average = {2:N2}", this.min, this.max, this.Average);
    }
}

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 21:24:52 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a serializable structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Collecting Performance Counters

/// <summary>
/// Collect stats. This method never returns (just kill the program).
/// </summary>
/// <param name="dictionaryPath">
/// The path to the dictionary.
/// </param>
private static void CollectStats(string dictionaryPath)
{
    using (PerformanceCounter processor = new PerformanceCounter("Processor", "% Processor Time", "_Total"))
    using (PerformanceCounter paging = new PerformanceCounter("Memory", "Pages/Sec"))
    using (PerformanceCounter freeMemory = new PerformanceCounter("Memory", "Available Bytes"))
    using (PersistentDictionary<DateTime, Sample> dictionary = new PersistentDictionary<DateTime, Sample>(dictionaryPath))
    {
        while (true)
        {
            dictionary[DateTime.Now] = new Sample
                {
                    ProcessorTime = processor.NextValue(),
                    Paging = paging.NextValue(),
                    FreeBytes = freeMemory.NextValue(),
                };

            Thread.Sleep(TimeSpan.FromSeconds(0.5));
        }
    }
}

Updated Wiki: SystemStats

laurionb — Tue, 15 Feb 2011 21:15:04 GMT

SystemStats Sample

This shows how to use a PersistentDictionary to build a simple application to track and report on some performance counters. There are two main parts:

A mode that periodically collects performance counter values and stores them.
A mode that queries the stored data.

The Sample structure

This program will use a PersistentDictionary to map a DateTime to a collection of samples. We will group the samples together into a structure.

/// <summary>
/// One sample of the statistics we are collecting. This struct is stored in
/// the PersistentDictionary, keyed by time.
/// </summary>
[Serializable]
internal struct Sample
{
    /// <summary>
    /// The processor usage.
    /// </summary>
    private float processorTime;

    /// <summary>
    /// The amount of system paging.
    /// </summary>
    private float paging;

    /// <summary>
    /// The amount of free memory.
    /// </summary>
    private float freeBytes;

    /// <summary>
    /// Gets or sets the processor usage.
    /// </summary>
    public float ProcessorTime
    {
        get { return this.processorTime; }
        set { this.processorTime = value; }
    }

    /// <summary>
    /// Gets or sets the amount of system paging.
    /// </summary>
    public float Paging
    {
        get { return this.paging; }
        set { this.paging = value; }
    }

    /// <summary>
    /// Gets or sets the amount of free memory.
    /// </summary>
    public float FreeBytes
    {
        get { return this.freeBytes; }
        set { this.freeBytes = value; }
    }
}

Updated Wiki: Home

laurionb — Sat, 12 Feb 2011 02:03:01 GMT

ESENT Managed Interop

ManagedEsent provides managed access to ESENT, the embeddable database engine native to Windows. ManagedEsent uses the esent.dll that is part of Microsoft Windows so there are no extra unmanaged binaries to download and install.

Project Components
The Microsoft.Isam.Esent.Interop namespace in EsentInterop.dll provides managed access to the basic ESENT API. Use this for applications that want access to the full ESENT feature set.

The PersistentDictionary class in EsentCollections.dll provides a persistent, generic dictionary for .NET, with LINQ support. A PersistentDictionary is backed by an ESENT database and can be used to replace a standard Dictionary, HashTable, or SortedList. Use it when you want extremely simple, reliable and fast data persistence.

esedb provides both dbm and shelve modules built on top of ESENT IronPython users.

Updated Wiki: ManagedEsentApis

laurionb — Sat, 12 Feb 2011 01:59:14 GMT

These links go to the MSDN documentation for the unmanaged API. The ManagedEsent version will be called Api.name and will have the same semantics. As much as possible, the managed version of the API will have the same arguments as the unmanaged.

JetAddColumn
JetAttachDatabase
JetAttachDatabase2
JetBackupInstance
JetBeginExternalBackupInstance
JetBeginSession
JetBeginTransaction
JetBeginTransaction2
JetCloseDatabase
JetCloseFileInstance
JetCloseTable
JetCommitTransaction
JetCompact
JetComputeStats
JetCreateDatabase
JetCreateDatabase2
JetCreateIndex
JetCreateIndex2
JetCreateIndex3
JetCreateInstance
JetCreateInstance2
JetCreateTable
JetCreateTableColumnIndex
JetCreateTableColumnIndex2
JetCreateTableColumnIndex3
JetDefragment
JetDefragment2
JetDefragment3
JetDelete
JetDeleteColumn
JetDeleteColumn2
JetDeleteIndex
JetDeleteTable
JetDetachDatabase
JetDetachDatabase2
JetDupCursor
JetDupSession
JetEndExternalBackupInstance
JetEndExternalBackupInstance2
JetEndSession
JetEnumerateColumns
JetEscrowUpdate
JetFreeBuffer
JetGetAttachInfoInstance
JetGetBookmark
JetGetColumnInfo
JetGetCurrentIndex
JetGetCursorInfo
JetGetDatabaseFileInfo
JetGetDatabaseInfo
JetGetIndexInfo
JetGetInstanceInfo
JetGetInstanceMiscInfo
JetGetLock
JetGetLogInfoInstance
JetGetLogInfoInstance2
JetGetLS
JetGetObjectInfo
JetGetRecordPosition
JetGetRecordSize
JetGetRecordSize2
JetGetSecondaryIndexBookmark
JetGetSystemParameter
JetGetTableColumnInfo
JetGetTableIndexInfo
JetGetTableInfo
JetGetThreadStats
JetGetTruncateLogInfoInstance
JetGetVersion
JetGotoBookmark
JetGotoPosition
JetGotoSecondaryIndexBookmark
JetGrowDatabase
JetIdle
JetIndexRecordCount
JetInit
JetInit2
JetInit3
JetIntersectIndexes
JetMakeKey
JetMove
JetOpenDatabase
JetOpenFileInstance
JetOpenTable
JetOpenTemporaryTable
JetOpenTempTable
JetOpenTempTable2
JetOpenTempTable3
JetOSSnapshotAbort
JetOSSnapshotEnd
JetOSSnapshotFreeze
JetOSSnapshotGetFreezeInfo
JetOSSnapshotPrepare
JetOSSnapshotPrepareInstance
JetOSSnapshotThaw
JetOSSnapshotTruncateLog
JetOSSnapshotTruncateLogInstance
JetPrepareUpdate
JetPrereadKeys
JetReadFileInstance
JetRegisterCallback
JetRenameColumn
JetRenameTable
JetResetSessionContext
JetResetTableSequential
JetRestoreInstance
JetRetrieveColumn
JetRetrieveColumns
JetRetrieveKey
JetRollback
JetSeek
JetSetColumn
JetSetColumnDefaultValue
JetSetColumns
JetSetCurrentIndex
JetSetCurrentIndex2
JetSetCurrentIndex3
JetSetCurrentIndex4
JetSetDatabaseSize
JetSetIndexRange
JetSetLS
JetSetSessionContext
JetSetSystemParameter
JetSetTableSequential
JetStopBackupInstance
JetStopServiceInstance
JetTerm
JetTerm2
JetTruncateLogInstance
JetUnregisterCallback
JetUpdate
JetUpdate2

Updated Wiki: Home

laurionb — Sat, 12 Feb 2011 01:42:28 GMT

ESENT Managed Interop

ManagedEsent provides managed access to ESENT, the embeddable database engine native to Windows. ManagedEsent uses the esent.dll that is part of Microsoft Windows so there are no extra unmanaged binaries to download and install.

Project Components
The Microsoft.Isam.Esent.Interop namespace in EsentInterop.dll provides managed access to the basic ESENT API. Use this for applications that want access to the full ESENT feature set.

The PersistentDictionary class in EsentCollections.dll provides a persistent, generic dictionary for .NET, with LINQ support. A PersistentDictionary is backed by an ESENT database and can be used to replace a standard Dictionary, HashTable, or SortedList. Use it when you want extremely simple, reliable and fast data persistence.

esedb is a dbm-like interface to ESENT for IronPython users.