Dealing with column default values

Apr 26, 2013 at 9:39 AM
Hi all,

I'm just starting to develop an application using the ManagedEsent library and have been testing some functionality using a copy of a "Windows.edb" database file. I've encountered a problem when attempting to retrieve three particular columns from one of the available tables - specifically the "A callback failed" exception is thrown when attempting to retrieve the column data.

In all three cases the column data type is "LongText" and the exception is thrown when I attempt to retrieve the column data using the "Api.RetrieveColumnAsString(...)" method. I've stepped through the code and other columns, whose data type is "LongText", are retrieved correctly so I don't believe that I have a fundamental problem with this data type - the only difference I've noted is that the three columns have a byte[] specified as a "DefaultValue" (as observed by inspecting the "ColumnInfo" object for each column), whereas all the other columns have this "DefaultValue" set to null.

So I guess my question is two-fold - why is this "DefaultValue" causing the failure of the "Api.RetrieveColumnAsString(...)" method, and how can I trap for this and deal with it accordingly ?

Phil H
Apr 29, 2013 at 8:16 AM
Hmm,

On further testing it appears that the "A callback failed" exception is thrown for these columns regardless of which "Api.RetrieveColumnAs..." method is used. Is this because these columns, for whatever reason, don't have a value and the "DefaultValue" needs to be used instead - this still appears to be the only difference between the three failing column retrievals, and the other working column retrievals ?

Phil H
Coordinator
Apr 29, 2013 at 8:44 AM
Correct, it's a 'User defined value'. It can be thought of as a 'virtual column', and when ESE tries to retrieve the value it instead calls a user-supplied function.

You could grab the DLL and copy it somewhere so that it gets loaded properly, but I'm assuming that this is just experimentation, in which case it may just be easier to use a different database altogether -- one that doesn't use callback functions. :)

-martin
Apr 29, 2013 at 9:55 AM
martinc wrote:
Correct, it's a 'User defined value'. It can be thought of as a 'virtual column', and when ESE tries to retrieve the value it instead calls a user-supplied function.

You could grab the DLL and copy it somewhere so that it gets loaded properly, but I'm assuming that this is just experimentation, in which case it may just be easier to use a different database altogether -- one that doesn't use callback functions. :)

-martin
Hi Martin,

Thanks for that information - so that'd explain why I get the "A callback failed" exception, since the callback function isn't there. So what is the byte[] value that the ManagedEsent library has stored in the "ColumnInfo.DefaultValue" ?

Although this is just experimentation at the moment, I will be wanting to parse the "Windows.edb" database eventually so I guess I need to hunt down the DLL where I can find this callback function and add a reference to it ...

Phil H
Apr 30, 2013 at 4:34 AM
Hi Phil,

The byte[] value is JET_USERDEFINEDDEFAULT structure. Please read this article on MSDN.

As you can see, it recommends specifying the absolute path to the DLL containing the callback. ESENT loads that DLL itself, adding a reference will do nothing.

You might get that exception because of the different security contexts: remember the windows search service is normally running in the local system security context.

Or, the DLL implementing the callback might interoperate with anything external to provide that value: another component of windows search, another windows service, device driver, LAN/WAN, etc.. In this case, I won't be surprised if you won't be able to use that callback in your application.
Coordinator
Apr 30, 2013 at 4:48 AM
First of all, I'm pretty sure that the Search Team frowns sincerely on digging through their database. And they reserve the right to change the internal format at any time.

You can examine the DLLs loaded by the Search process.
You can also use procmon.exe to see which mysterious DLLs are attempted to be loaded by your program, specifically at the moment you try the RetrieveColumn call.

If by 'adding a reference to the DLL' you mean that it's a CLR DLL, I'm pretty sure it would be written in native code.

(Hmm, I can't remember if I wrote that bit on MSDN or not... I must be getting old. :)

-martin
Apr 30, 2013 at 9:28 AM
Thanks for all all the comments, I'll take a look at the JET_USERDEFINEDDEFAULT structure and see if I can make any useful sense out of it. I see what you mean about the different security context - it may be better to simply ignore (or flag) these columns, but do nothing with them. In fact I should've thought of martinc's comment re:native code - it makes sense that this would be the case, in which case there is very little likelihood of being able to integrate any such DLL correctly.

With regards to martinc's comment re:the Search Team frowning sincerely on digging through their database etc; I have no issue with them changing the internal format, after all they've done it a number of times already when moving between different versions of the OS :-) As for digging through their database, the reason for my interest is the potential forensic information that may be contained therein (e.g. filenames that have been indexed, etc) - I'm certainly not planning on creating anything that interferes with the Search process, I'm purely interested in being able to identify forensically relevant information that may be contained within the database :-)

Cheers,

Phil H
Coordinator
Apr 30, 2013 at 11:06 AM
You mentioned forensics. If you have an actual criminal case, then you really should contact Microsoft Support. They offer forensic services for Law Enforcement purposes.

To be honest I have no idea how much they charge. It might be a few hundred dollars, which in a criminal case, you'll hopefully have budget for. There is also a chance that there is no charge.

-martin
Apr 30, 2013 at 2:49 PM
Hi martin,

Sorry I should have explained myself a bit better - I actually work for a Forensic Services provider in the UK, so we are actually involved in investigating criminal cases for Law Enforcement etc. Our interest, particularly in the "Windows.edb" database but also other relevant ESE databases, is in being able to examine these databases to produce evidence in a suitable format. I believe that Microsoft do indeed have their own forensic team, but we wouldn't typically engage them ourselves :-)

Cheers,

Phil H
Coordinator
May 2, 2013 at 10:53 PM
Now I'm picturing you in Thames House in a Spooks episode, or working at Bletchley Park or GCHQ or something, all James Bond-like. :)

You should try the procmon route to figure out which DLL.

By the way I asked around for another person on another issue, and the answers were:
'IIRC, Microsoft has a specialized team that deals with Law Enforcement Support. '
'The special team that assists law enforcement with these questions is generally CSS, after law enforcement has gone through the LE portal run by the DCU. '

And the glossary:
To translate:
CSS -- product support
LE -- law enforcement
DCU -- ???

But MS in UK may have different policies and procedures.

-martin