Santosh Benjamin

BTS Diary 10 - Catastrophic exceptions near the finish line

Well, its nearly done and dusted now. Touch wood, we should we going live soon. The experience with deployment has been nothing short of annoying (to be polite!!!) When we need to deploy new versions of orchestrations, we run a batch file (copied and customized from the samples in the whitepaper on Developing Integration Solutions with BTS 04, which again derived from some SDK samples). this file attempts to stop all the orchestrations and then stop the ports. For ports and locations that utilise custom pipelines we use a dummy binding file which replaces all these pipelines with default pipelines because for some reason, manual intervention is needed when undeploying artifacts with custom components. We then remove the ports and undeploy the MSI. In this story Its the inconsistency in the experience that troubles me. Orchs sometimes refuse to stop, ports refuse to stop, they refuse to be removed. Why? Heaven knows. We get some weird hexadecimal errors from the VBS files. So we then have to forcibly remove all the ports and orchestrations manually, trying various routes and options to do so. It gets removed eventually but not without a fight. I dont know if others have similar experiences or if we are just jinxed. I also hope the clients dont come up with requests for new features (!!!) or this cycle is going to repeat very very often and lead to a lot of tears and hair loss.

But back to the title of the post. The core functionality of our system is a data load engine that takes data in from nearly a 100 suppliers  (ok, so not all of them are on board yet) in different formats and then transforms it into a common schema and then loads into a db via a web service. Other business processes then take over and work on the data.  The file size is critical because theres a webservice at the end which does the actual data loading into a custom database. So no matter how 'big and powerful' biztalk is, Http and SOAP can only handle so much. Secondly we found that if we put 10-15 files in the recieve location at one time, BTS tries to grab all of them at one time leading to deadlocks in the database. So when we get the first load of data from suppliers which could be around 100-200MB we run it through a custom application which splits the files into 'manageable' sizes and then pumps one by one into the recieve location.

In spite of all this external debatching, We are very often seeing Out of Memory exceptions and  'Catastrophic Exception' ocurred in the event log. I checked out a couple of articles where the authors had related problems but they relate to huge messages, over a few hundred MB. In our system, the max size is around 35MB (in a text file) corresponding to approx 80MB when mapped to an XML schema. This happens even on servers with 2GB or more RAM which are dedicated to running Biztalk alone!

Im fervently hoping BizTalk 2006 will have a better file handling and deployment story.

Oh, let me end on a positive note. Tom Hollander, the product manager of the Ent Lib team came across my posts where i complained about the Ent Lib logging stuff and wanted more detail so his team could investigate.!! Wow!! Talk about responsive!!.