This project has moved. For the latest updates, please go here.

Proper way to handle JET_err.DatabaseDirtyShutdown

Sep 7, 2011 at 7:14 AM

I caused a dirty shutdown by purposely killing an application that was constantly updating rows. 
I am looking for pointers on how to properly handle the JET_err.DatabaseDirtyShutdown exception when my application restarts.

I looked at EnsureDatabaseIsCreatedAndAttachToDatabase from ayende (https://github.com/ravendb/ravendb/blob/master/Raven.Storage.Esent/TransactionalStorage.cs), but for some reason this code gives the same JET_err.DatabaseDirtyShutdown exception when trying to attach with AttachDatabaseGrbit.DeleteCorruptIndexes using the recovery instance/session.

@Laurion: are there any "best practices" when it comes to which steps to make in order to recover your own database? I always imagined it would be enough to initialize esent with recovery = true (after setting correct basename, folder paths, etc).

Sep 7, 2011 at 4:37 PM

You are correct when you say "I always imagined it would be enough to initialize esent with recovery = true (after setting correct basename, folder paths, etc)." My guess is that one of two things went wrong:

  1. You were running with recovery=false when you killed the application. You always have to have recovery turned on for crash recovery to work.
  2. You didn't correctly duplicate the log path when initializing.

There is no difference between initializing  esent for recovery and normal initialization, you should be using the same codepath.

Sep 7, 2011 at 8:28 PM

Thanks again Laurion! 

After starting over with a fresh database, I can no longer make the JET_err.DatabaseDirtyShutdown exception occur after killing the process.
I did notice that the next init call takes a bit longer (so it seems there is some recovery/log replaying going on during normal initialization as you stated). Are there any way I can detect if the init call also performed a recovery operation (for logging purposes)?

I have another question related to ayendes implementation - what kind of situation is he trying to correct in the EnsureDatabaseIsCreatedAndAttachToDatabase  function?

According to the docs you would use AttachDatabaseGrbit.DeleteCorruptIndexes (http://msdn.microsoft.com/en-us/library/gg294074(v=EXCHG.10).aspx) when you want to ensure that esent checks and fixes potential corrupted indexes (which would stem from different unicode normalization across OS versions).

What kind of situation will make the DB be flagged as dirty shutdown where you also can "fix" the db by re-attaching to it with AttachDatabaseGrbit.DeleteCorruptIndexes set?

 

Sincerely,

Paal

Sep 8, 2011 at 5:53 AM

The way ESENT is structured is that recovery init always performs recovery, it is just that sometimes recovery doesn't have anything to do. The idea is that the application doesn't need to care whether it was cleanly shutdown or not -- I am actually in agreement with the people who argue that applications should always exit without proper termination, so that the recovery path is the only path :-)

Initializing ESENT when there are unicode sort order changes is incredibly complex and the code you are looking at isn't actually correct. If there is a unicode sort order change you'll get a specific error at attach, not JET_err.DatabaseDirtyShutdown. That area isn't handled very well and because the debug hooks we used to test sort order changes at Microsoft aren't available in the released code there isn't an easy way for applications to test. If you are going to have indexes over unicode data then taking Ayende's RavenDB approach of dump/load is the simplest and most reliable.

Sep 8, 2011 at 7:52 AM

I can understand the argument about never properly terminating - how else can you make sure the recovery process works flawlessly? :)
I feel very comfortable knowing that if the DB breaks (e.g my application is not able to attach/init) it will most likely be caused by a hardware fault (or really bad programming on my behalf).

Just to be sure I've understood correctly, the code example from RavenDB should instead catch JET_errPrimaryIndexCorrupted and/or JET_errSecondaryIndexCorrupted to handle OS version changes (where dumping the indexes and recreating them can be considered an "upgrade path")? 

If your application still encounters  JET_err.DatabaseDirtyShutdown upon attaching - this would basically mean there is no hope left? (assuming you are attaching to your own database, using the correct basename and log file paths).