CF_MKVDK: Programmatically Index Documents with Verity's Utility mkvdk
Jan 14
posted by Steven Erat in
Flex, Linux, Quality Assurance, ColdFusion, Travel, Macromedia
| 7 Comments
Unofficial Workaround for CFMX Crashes When Indexing PDF documents on Solaris.
If you've ever been interested in checking out the Verity tool "mkvdk" or if you are currently experiencing JVM crashes when indexing PDF files on Solaris with CFMX 6.0 or 6.1, then you may want to have a look at a custom tag
cf_mkvdkcreated as replacement for using CFINDEX.
In the ColdFusion documentation the mkvdk utility is described, briefly, as:
"The mkvdk utility is an indexing application, provided with other Verity utilities, that you can use to create and maintain collections. It is a command-line utility that you can use within other applications or shell scripts to provide more sophisticated scheduling and other capabilities."
#1 by joshua frankamp on 7/27/04 - 4:11 PM
[P]First want to say, this is pretty cool.. looks to be some fun hacking. I have a question... I am working with verity K2 server on Solaris 8. I seem to be crashing the server with just several successive searches, suddenly JRun is dead and k2server is still happily processing. Max threads on CFMX are 20, Listeners and Threads on k2server.ini are 25 (should be plenty, but ive tried lots of settings for these to no avail).[/P]
[P]We do have a lot of pdfs... Does this indexing `crash` corrupt the data in our collection somehow, so that as I search it I am seeing a ghost of this same problem?[/P]
[P]Any ideas you have would be helpful,[/P]
[P]Joshua[/P]
#2 by Pushpa on 11/12/04 - 5:00 PM
I noticed that sometimes the collection folder is created and sometimes it`s deleted.
I`m not sure what`s going on here.
Can you please let me know.
Thanks
#3 by Steven Erat on 11/12/04 - 5:24 PM
I really don`t support this tag. Since you have the source, you`re free to rewrite it to suit your needs.
In short:
- its called as a custom tag
- it does a recursive directory search for matching file names
- it creates a specially formatted bulk file with Verity input information
- it executes mkvdk utility and passes the bulk file as input
The mkvdk utility can create a Verity collection file structure using the -create switch but ColdFusion won`t know about it. The way I wrote the custom tag, it tries to create a collection using CFCOLLECTION so that ColdFusion then knows about it and it shows up in the CFAdmin.
I recommend that you consider modifying the code to alter any behavior you currently see.
By the way, Blackstone (CF 7) will come with a new version of Verity that should largely eliminate lingering problems like this because Verity will no longer use Native Code that may be prone to crash.
#4 by Tony on 4/5/05 - 1:53 PM
Do you have any more status on this problem? We use Verity k2e to index, not cfindex. But we do get at least 5 hotspot crashes everyday on random searches to these collections using cfsearch.
#5 by Steven Erat on 4/5/05 - 2:26 PM
While you might be using the K2 server with ColdFusion, the K2 server only does searching, not indexing. Indexing in CF5, CFMX 6.x, and CFMX 7 is handled by the utility mkvdk which is called by an intermediate libCFXNeo.so. The JVM runs java code which then calls libCFXNeo.so as "native code", and that in turn calls mkvdk.
After I got started building a custom tag to call mkvdk directly, it looked useful for overcoming the specific problem of the JVM crashing *when indexing pdf files in a loop*. That was the one known scenario where calls to Verity would crash the JVM.
So if you`re not indexing pdf file in a loop, then you have a different problem. It could be this one which occurs on Linux since the 1.4 JVMs.
Hotpot crashes on Linux
#6 by Steven Erat on 4/5/05 - 2:27 PM
#7 by Daniel on 1/2/08 - 2:20 PM