June 2006 - Posts
Ordered Delivery: So, continuing on the issue of ordered delivery and queuing, one important point which Tomas pointed out in the newsgroup is that it is not enough to just send to the queue whether it be MSMQ or MSMQT in sequence, but the orchestration that subscribes to it must also be a singleton if we want to ensure the sequencing. This makes sense because if we put a ton of messages into MSMQT or MSMQ then automatically a lot of subscribing instances are going to kick in to consume the messages and if in a set of messages, one instance is slightly slower to execute than another (say, a particular branch of the orchestration has been picked which the other instances didnt get), then it might happen that a message intended for later appears at the destination first. So if you wanted to keep inserts and updates in sequence and the update reached first because of the processing situation, then the destination system could throw an exception. The singleton would stop this, but we would sacrifice some processing speed. Without the singleton we would have to put in custom logic in an orchestration to handle the exception. For example, if we are sending an update or delete message and we recieve an exception, we could invoke a branch which set a delay interval and tried again for a few times and if the final attempt didnt work, it could put the message in a business exception queue which should be monitored and picked up. It all depends on how many messages are being pumped in at the source. If that is controlled then singletons and sophisticated exception handling is not needed. If a small number of messages are flowing the singleton could be good enough (assuming you do face situations where the messages have overlapped). If there are hundreds of messages coming in at source, then throughput becomes extremely important and the exception handling would be the better approach [if you are aware of something even better, then do let me know]. I'll prototype this and let you all know what I find. In order to avoid changing the design of the orchestration later, i am leaning toward the exception handling approach now (just to be safe).
SSO in distributed environments. Heres a tip in case you encounter this annoying SSO error message. We finally got round to deploying the code into DEV and immediately hit a snag. We had been using MSDE on our workstations and giving the ASPNET user the needed access to the Biztalk databases. Once we got to a distributed environment, of course the ASPNET user on the app server couldnt be given any access to the database server. This access is very important to us because we have orchestrations published as webservices. We hit the familiar error message saying ; SSO AUDIT GetConfigInfo failed. Access Denied. etc. We used impersonation and gave the domain user access to the database and made them members of the Biztalk groups and all that and eventually when we set the user to be a member of SSO Administrators and SSO Affiliate Administrators on the application server, the problem got solved.
Workgroups: Its amazing how no matter how many times you install Biztalk 2004, theres always a problem in configuration. 2.5 years ago when i first started with it, i spent nearly a week installing and uninstalling and deleting databases for the config to work. About a month ago i managed to get it to work with only 1 rollback. That was my best experience so far. Yesterday, I was not so lucky. My previous laptop had been stolen from my desk at the office and so i was given a replacement (just 512MB RAM mind you, in contrast to the 2GB on the old one.. sniff!! sob!!) . I managed to install VS.NET and SQL and then when it came to Biztalk i dredged up the old articles from memory and set the registry permissions for EnableAuthEpResolution, enabled RPC Endpoint authentication etc so that SSO wouldnt give me the stupid "The Enterprise Single Sign On service failed to start" error that plagues folk when they install BTS. I also made the BiztalkUser account that runs all services a member of the Admins on the box. Drastic measure, i know, but sometimes when you just want to get the dratted thing installed on a laptop, priveleges dont really matter. Besides, i dont think there is any accurate doc on MS that says exactly which accounts need to be setup for development and what priveleges they need (for BTS). Anyway, i thought i had been smart. Boy, was i wrong!!! I didnt get the expected error message but got another one "Failed to generate the master secret". This plagued me for the rest of the evening. Finally i didn what this article says and since i got the return code =2, i removed the laptop from the domain and then the configuration worked!!!
I think its quite sad that MS doesnt have another SP or hotfix with a set of scripts that can do all these settings before you start configuring Biztalk 2004. MS has moved onto Biztalk 2006 and Biztalk2006 R2 and planning for vNext, but a lot of people are still on Biztalk 2004 for various reasons and we would appreciate the install and configuration experience being made just a little bit better till the day we can move to 2006 or whatever comes next. I installed 2006 on a spare workstation the other day and there were no errors whatsoever. (I did use a account with admin priveleges again) and it even connected to the net to download missing prequisites and automatically installed them. Is it so difficult to patch the 2004 configuration engine to do something similar? Or even if it cant do this, why not give us the ability to simply SAVE the configuration settings which we so laboriously set up (user accounts, database connections etc), before proceeding to execute the configuration so that if it rolls back , we can fix whatever is needed, reload the configuration and go on instead of having to start from scratch? I hope someone from MS is listening but i fear its too late. They didnt care to do it these past 3 years and i doubt they are going to start now.
If you havent started Biztalk yet, i'd advise you to avoid v2004 and go straight to 2006. The install experience alone is worth the switch.
Some more information for those who have been following my posts on this topic.
MSMQT : It appears that the MSMQT behavior is by design. Since it is a pseudo queue and really puts stuff into the messageBox, the basic msgBox principle has to apply (ie) a subscriber has to exist. So a dummy send port sending stuff to a file is acceptable even if it is stopped or unenlisted temporarily. Tomas Restrepo, an MVP on the newsgroup suggested i consider moving to MSMQ instead and use the MSMQ adapter since that is the MS recommendation for doing message queueing. Fancy that!! they release a pseudo queue and ask us to consider using something else!! Of course, there are pros and cons for each. For a comprehensive discussion on the history of MSMQ adapters in Biztalk, refer to this article . Anyway, the key point for me is that there is no guarantee of ordered delivery and in this phase of my project we have a service that pumps out customer data into a queue which is then picked up by an orch and sent to the hub which does the dispatch to other systems. Since the customer data involves inserts, updates, merges etc, the sequence is vitally important otherwise we would end up with unnecessary exceptions at the other end. (We may send an update message to the target system before sending an insert and so on). It was also suggested that i consider piping the MSMQT files to a folder while an update is going on and then resubmitting the files once the patch/redeploy is complete. A good suggestion, but the FILE adapter doesnt guarantee sequencing either. So im going to have to either write a custom adapter or work out some other tool which submits the files to the receive location one after another in the correct sequence. I did write something like this for my previous project (the controlled submission was to avoid locking in the database) so perhaps i can reuse that here.
BizUnit, XML Validation and AltovaXML: Turns out that there are problems in XMLValidation when using BizUnit. It cant handle validating against a collection of schemas even when all the 'included'/imported schemas are in the same folder. Additionally, the XPath statements are constrained to those that can return nodesets only (in BizUnit 2006 there is an enhancement which works around this). So we had to look outside of BizUnit. We considered writing our own custom steps, but for the present the time element is critical and although that is a good long term solution, we didnt want to go down that route today. Fortunately, BizUnit has an inbuilt 'extensibility' mechanism or in other words, an escape route, in the form of a DotNetObjectInvoker which allows you to invoke any dotnet type that can take input parameters and return a result. So all that was needed was the requisite classes which could then be invoked from BizUnit. For XPath we wrote a helper class to evaluate and return expressions of integer, string and boolean values. For the XMLValidation, we found that Altova provides the AltovaXML 2006 library free of charge for developers ( i need to check the production license, just to be safe) and that has .NET interfaces as well as Java, COM and XSLT interfaces. So we wrote a wrapper around it and called it from the DotNetObjectInvoker. It works like a charm. So we are going to make extensive use of that in our tests. We also found that there is no 'CommandLineStep' which could be used to run a Shell exec on some command line statementas but that should be trivial to write (a bit like the NAnt 'exec' task) .
My talk at the VBUG last week went off well. Thanks to the people who attended for making it a very enjoyable session (45 minutes of Q&A after the session must be some kind of record!!) and of course to VBUG for giving me the opportunity to speak. I look forward to the next time and im considering something in October, topic to be decided later. Anyway on to the subject of todays post.
MSMQT Tip: I found something about MSMQT that changed my understanding of queues or at least MSMQT in particular. The way i understand queues is that you can merrily post to them and the subscriber at the other end could be down and no one will care and when the subscriber comes up again, it picks up the messages. So accordingly we set up the MSMQT receive location and posted a message to it. Whadday know!!immediately Biztalk complained that it didnt have a subscriber. This baffled us for a while. So we created a send port which subscribed to the recieve port and dumped the message to the file. Of course this worked, but this is not the point of a queue! More experimentation followed. We stopped the queue leaving it enlisted. No proble and no error messages. As soon as it came back again the messages got dumped. Then we stopped the queue and unenlisted it.Strangely no problem at all. Once again as soon as it came back up the messages got dumped!! It appears that MSMQT wants to have seen at least one subscriber at some point of time in the beginning of the queues lifetime and then it doesnt mind if the subscriber is down. Isnt that odd?
The thing is , I will have an orchestration subscribing to the queue shortly but the queue is there so that i can also avail of the disconnected feature and then undeploy the subscriber for patches and bugfixes. But this also means that i cannot do a complete rebuild of all the ports etc because if i lose the dummy send ports or the orch receive locations then im stuffed. The messages will get lost. I will have to do a complete new mechanism for bug fixing which doesnt involve chopping out and rebuilding ports along with orchestrations. Im going to post this on the newsgroup and see what people say but in the meantime , keep this in mind when using MSMQT.
BizUnit tip: Now on to BizUnit and once again, i'll keep you updated with what i find if its a bug or feature. It looks like we canot use the SOAPHttpRequestResponse step to post to an untyped webservice. We have an orch published as an ASMX and it takes an XmlDocument as input. No matter what InputTypeName we declared (in the BizUnit test case), BizUnit wouldnt hear of it . It kept complaining that it could create the necessary type. So we had to resort to NUnit instead and just loaded in an XmlDoc from file and sent the nodes across the wire. We'll use BizUnit to test and validate the output file once we save the response locally.
Theres also one more thing about BizUnit which i think is dodgy. You cannot post a complete SOAP request to it (for a typed webservice). I cut and pasted the SOAP request from the ASMX helper page but it just wouldnt work. What you have to do is pick up a proxy object and then serialise it , copy the serialised XML and then send it to the SOAP end point. Thats really annoying as i was hoping to avoid generating webreferences and proxies. Moreover, if the ASMX page shows the structure of the SOAP message, why cant BizUnit pick that up? I can understand if we only remove the SOAP envelope and body and if BizUnit adds that dynamically, but the entire input message structure was kinda funny (compared to the ASMX page that is). Really weird, but i will discuss this on the GDN workspace and post my findings.
Well we had the most awful time with this the other day. I posted this on the biztalk.general newsgroup but eventually solved the problem myself. Heres what happened.
We had two orchestrations published as webservices. (Using Biztalk 2004 on XP SP2) One was typed and the other untyped (ie) consuming an XmlDocument at the recievePort and assigning it to a specific message variable internally. The untyped one worked fine when called from another orchestration and from an NUnit test. The typed one however always threw a weird error message saying " Server was unable to process request. --> File or assembly name FileName .dll, or one of its dependencies, was not found." . We attempted the following
(a) I looked at the support article for the same at http://support.microsoft.com/?kbid=910295 but to no avail. I gave all possible users permissions on all temp folders, but the same message appeared.
(b) Both webservices had anonymous access enabled and i set the ASPNET and Isolated Host users to have permissions to the temp folders and the databases. (I even temporarily let the ASPNET user run as SYSTEM :-)).
(c) I tried adding some code to the asmx.cs to write custom statements to the event log but that didnt work either. It seemed that the web service was just not being called.
(d) found an article which said that Proxy projects that have names more than 61 characters long cannot be compiled (weird huh). But even shortening the names to well below 61 didnt help.
(e) enabled detailed error tracing but this was of no use since the webservice was not being JIT compiled correctly (thats the reason for the weird error message)
Finally we had a breakthrough. We first changed the input operation to accept an untyped message. The system appeared to run for a longer time but returned the same error. While looking at the custom logging files we had setup (using EntLib) it appeared that the orchestration was indeed being invoked by the webservice but the system still threw the exception. Looking at the stack trace we found something about a method which has the word 'response' in the name. At that time this didnt ring any bells, but in desperation we changed the return type to XmlDocument and the wretched thing started working!!
It could have just been the response message wasnt being constructed correctly, but all we do now is construct a typed response message and then assign it to an XmlDocument and send it to the response port.
I cant understand it. Typed messages are the bread and butter of Biztalk and untyped messages are not frequently used. So for the system to complain about a typed message is very strange indeed. Anyway, this is a tidbit i thought i'd share in case someone else comes across this problem anytime. Im hoping to find some time later to try and reproduce the problem and do some more detailed debugging. Its a pity that orchestrations are so difficult to debug (even with the orchestration debugger).
Speaking at a user group: I've been invited to do an Introduction to Biztalk at the .NET Exchange User Group in London next week. The speaker who was supposed to do this dropped out and as i had volunteered earlier to share my experience with BTS, I've been asked to do the honors. Gotta get the old slides polished up and get a demo working.