July 2006 - Posts
Just finished watching a webcast on
Enterprise Single Sign-On integrated with Microsoft BizTalk Server 2004
and Microsoft Host Integration Server 2004 . If you havent already seen that or arent familar with how SSO works and how AD and password synchronization can be used together with SSO, then check this out. Its really well presented with good demos. Since the transcript and PPT are also available if you dont have time to watch it all, then read the transcript. The SSO management story has improved a good deal in BTS 2006 and this should make the solution even more compelling. It should also be interesting to see how far they have progressed in linking this with ADFS and MIIS now.
We've been considering a sort of Customer Data Hub to work along with the integration hub in our platform and i came across an excellent article by Chris Mc titled "Customer Data Integration: What the EII, ETL and DQ Don’t want you to know". In the introduction, he says
"In the last few years, enterprises have added several tools to their IT data integration toolbox to improve the return on their CRM investments. Yet, most of these companies have not achieved a simple goal: to create reliable, unified views of their customers – aggregated across data silos – and deliver these to all customer-facing applications in a timely fashion. Recently, companies have turned to three common technologies to create solutions for customer data integration. These are data movement tools such as Extract-Transform-Load (ETL), data query and aggregation tools such as Enterprise Information Integration (EII) and Data Quality (DQ) tools. However, what the tool vendors aren’t telling you is that these tools are woefully inadequate for developing a reliable Customer Data Integration (CDI) platform. "
Chris then goes on to explain the key considerations involved in building the Customer Data Hub and where EII etc have traditionally fallen short. His discussion struck a chord with me as i have just started to pull together some of the requirements to look at product/bespoke options. For veterans in the field, there may not be any earthshaking revelations here, but its a nice compilation of things to think about.
He didnt explicitly say anywhere who he works for but there are a couple of posts about Siperian and it appears he works for them. However, much to his credit, he does not actually plug the Siperian product anywhere (but if you look through the article carefully, you will find that the proposed architecture for a 'comprehensive' CDI Hub matches what Siperian is offering)
The resource library on the Siperian site is also very informative and I'm definitely going to look into their product for our project. If you are considering such a solution in your architecure, take a look at the article. Even if you want to build a bespoke solution, this has some good points.
Just came across this article on SOA Antipatterns at InfoQ. This is a very well written article. Steve used to be our Chief Architect a while ago before he moved on to join CapGemini.
InfoQ Blurb: SOA Expert Steve Jones from CapGemini provides a hands on look at SOA Antipatterns and a list of ways your SOA project can go wrong. This list includes signs that these problems are cropping up as well as what to do when you see them happening.
Check it out. It definitely gave me some food for thought.
One of the things I've been meaning to do for a long time is to get into some book reviews. I have a rather extensive library of tech books and loads of them are in the "to-be-read" queue and it seems that the best way to cover them is to put up a review. Another thing which i can do is review stuff directly from the publisher which would help fellow readers.
So I've started on the latter approach for now and the first book I'm going to review is the
BPEL Cookbook: Best Practices for SOA-based integration and composite applications development
The publishers are sending me a copy of the book soon and i plan to get through it as soon as possible. My main aim is to understand how SOA applications are done in different technologies and although my main focus is Biztalk and .NET development , SOA transcends platform and im sure i can pick up quite a few tips to apply to my integration project as well.
I've also got quite a few books loaded up into my DevX Safari bookshelf dealing with SOA. They are
(1) Enterprise Service Bus by Dave Chappell
(2) Enterprise SOA: Designing IT for Business Innovation - By Thomas Mattern, Dan Woods
(3) Enterprise SOA: Service-Oriented Architecture Best Practices - By Dirk Krafzig, Karl Banke, Dirk Slama
The third one has got rave reviews on Amazon and i'm looking forward to getting into it , well, as soon as i finish the BPELCookbook. I must say, the Safari bookshelf is really useful and its caused me to buy even more books cos i dislike reading online!
Anyway, I hope these reviews will help you choose the books that you want to buy. Watch this space for the first review.
Whew!! It took an entire evening but i finally cracked the mysteries of the BizUnit context and how it can be used. I even wrote a couple of my own custom steps to manipulate context. Writing BizUnit custom steps is really easy. The ITestStep interface and the Execute method are all you need.
What really took me by surprise is the number of bugs in it. Take the XmlContextLoader for example. This step allows you to extract data from an Xml file ,using XPath expressions, and store them in the Context. The step will not work out of the box because a stream which contains the data is being read into a string, and then the stream is being used to load up an XmlDocument without rewinding it first. I also fixed a couple of others including the weird "Param" typo that i blogged about earlier. Just after i had finished all this i checked the GDN workspace and found a reply to my posts from Tanveer Rashid which was quite helpful, but then again, i had solved the issue by that time.
So, Im planning to write a detailed document covering the use of Context and perhaps extend it to be a general BizUnit tutorial. I've reworked the documentation thoroughly especially in the area of the context load steps and im pretty comfortable that its much clearer now than earlier. I've also written more tests and documented the existing ones so folk can get an idea of whats available. I'm also considering starting up a BizUnit extensions project on the lines of NAntContrib to add more custom steps for the community (if Kevin doesnt grant me permissions to join the existing workspace, that is). Let me know if this is of interest and if there are custom steps that you would like to see. I should have the Context tutorial up in a couple of days max.
First of all let me state that I admire the effort Kevin Smith has put in to BizUnit 2.0 and 2.1 and I dont really want to grumble but frankly I'm getting quite frustrated with BizUnit now. [Btw, Im not just going to whinge about it ... I've put in an application to join the workspace and intend to at least fix the typos and put some documentation in].
A question of 'Context' : It started off quite well and its really simple to write the basic tests like checking for files, validating their contents and even manipulating database contents. But now that we have got a battery of 25 or so tests, maintenance has become a nightmare. For instance, in one of our tests we have about 5 or 6 different steps that all work on a particular folder and some files in them. The tricky bit is that the file path and name are duplicated in 5 places and one change requires a huge amount of work to go and change all those occurrences.
I was thinking that a concept like NAnt properties would have been so useful here. We could define all these common variables in one place and use them across the scripts. I get the feeling that the BizUnit Context class might prove to be handy to do this kind of thing, but the documentation is so poor, i cant find any info how to use it. Worse, the GDN community appears to have gone to sleep in that particular message board. I've experimented for days on end, but no joy. The author posted a rather cryptic one line example (to an earlier query) saying that we could do "context.Add(key, object)", but neglected to say where you would define and instantiate the context object. When i tried to use it, the system always complained of using an unassigned context variable. Moreover, defining it in code is not of much use because in BizUnit the meat of the system is the Xml test case so we need to have an ability to assign those variables in Xml. The doc for the Context Manipulator step seems close to what we need, but again, agonisingly, the example is not complete enough to make sense. There isnt any example to say how we can pick something from an existing context in one of the test case xml steps and use it (like the way we would use ${property} in Nant). What makes it even more confusing is that the source code shows context being passed between various steps but in BizUnit you dont invoke individual test steps in the CS file. You simply point the BizUnit constructor to the XML file and then say RunTest. Thats it.
Then there is the Text Context Loader with an absolute gem (sic!) of a comment. Heres what it says in the remarks on how to use it . "Here's the SearchString, but then skip this bit and return: TargetFindString" Huh? Say that again?!! I've gone over that several times but it doesnt make any sense. The same comment is carried over to the 2006 version including a really problematic typo that I'll explain below.
The "Parameter" typo: This little beauty wasted 3 hours of our time one day. BizUnit provides a sort of extension mechanism called a DotNetObjectInvoker which allows you to call any of your components to do stuff you want provided it can accept inputs and return an output. Now when you use this you define a set of parameter elements in Xml and BizUnit picks that up, reflects over your type and executes the method. The funny thing is, the Xml element is spelled "paramter" in the xml sample and the code. So, one of our developers had not noticed this and had used "<Parameter>" in his Xml file. Of course, it wouldnt work. It kept throwing "Object reference not set to instance of an object" all the time. We couldnt figure it out for a long time because we had unit tests for that component and they worked fine. Finally i spotted the reverse typo. !! What makes me rather puzzled is why this has not been corrected in v2006. Kevin claims that there have been over 2000 downloads of this tool so far, surely someone should have come across this error? There are quite a few other typos too (such as the SoapHttpRequestResponse step) and there are quite a few bugs still open on the message board.
As i posted sometime earlier, we have Xml schemas which make extensive use of includes and imports. This cannot be validated by the inbuilt XmlValidator in the BizUnit system so we had to use the DotNetObject invoker over the free AltovaXml library to do what we needed to do.
Another limitation is that there seems no obvious way to test untyped webservices. I accept that this is not usually the norm (I have come across systems with the single catch-all ProcessMessage(string input) signature and hot debates rage on whether this is good practice or not, but i wont go into that here now) but we do have some internal webservices (orchs published as webservices) which take XmlDocuments as input. The SOAPHttpRequestResponse step isnt of much use here .This step requires you to define what the type name of the input message is and the WSDL doesnt show anything useful (since its only an XmlNode). So, to work around this we needed a combination of BizUnit and standard NUnit /.NET. We defined the constructor explicitly to use the TestGroupPhase.TestSetup enumeration and pointed that to an Xml file purely meant for the folder/db cleanup etc. Then in the test we loaded an xml doc from disk, put that into an XmlNode and called the webservice using its proxy. After that we pointed bizunit at the xml file and let it run the rest of the steps. This isnt an elegant solution because it disconnects the xml from the rest of the test (in the source code) and makes it difficult to read, but for now this will have to do.
I do have some options up my sleeve though. Either I write another custom step which can call untyped webservices, or use reflection and pass a reference to the proxy object into my custom class or thirdly, look at DynWSLib which is a .NET library to dynamically call webservices at run time without needing proxies, written by the gurus at thinktecture. In the spirit of not reinventing the wheel, im going to look at option 3 first and see if i can incorporate that, failing which i will do option 2 and then option 1.
Extending BizUnit and the possibility of a GUI: I was reading up the blogs at Conchango and it looks like the folks there have been extending Bizunit quite a bit but it looks like theres a lot of IPR involved so probably no hope of that being released for general consumption. I also read somewhere that in one company they defined the test cases in Xml (so business folk and testers could define test cases) and then generated the xml for the test cases from there. This sounds like a good idea. However, IMO, what would be most useful is a nice GUI which allows you to define test cases and reads prewritten test cases and allows them to be edited. This could look like the Ent Lib configuration wizard. If i can squeeze some time out now, i may just dive into this GUI option and see what i can come up with.
This particular serialization problem took 2 midnight sessions to sort out. We had some untyped orchs published as webservices inside the service boundary and we were building a strong typed ASMX gateway as the service boundary. We used WSCF to generate the ASMX and used all the schemas that were inside the orchestrations so that we didnt have to maintain two different sets of messages. We then came across 2 problems, the first one more easily solved than the other. I'll explain them below
(1)Duplicate namespaces: When you define a set of schemas in Biztalk, it is perfectly acceptable to define a common target namespace for example http://mycompany/schemas/services/customerservice and then to have different messages with specific root elements such as CreateCustomer, UpdateCustomer and so on. When biztalk recieves the message and validates them and works out the subscriptions, it adds the root element onto the target namespace making it http://mycompany/schemas/services/customerservice#CreateCustomer (note the # before the suffix) and this is enough for it to resolve most schemas. However, if you put all these Create, Update and Delete methods in your ASMX, then when you add a web reference, the .NET system crashes complaining of duplicate namespaces. Actually, the WSCF tool complained about duplicate namespaces even while creating the ASMX so we bypassed it and coded the ASMX ourselves (not realising it was right in complaining). This stumped us for a while, but it looks like this is default behavior in .NET and Biztalk has the ability to handle it but not the normal .NET framework which is fine. So we worked around it by changing the targetNamespace in all our schemas to have some extra phrases such as create, update and delete. Of course, if you put these kind of suffixes into the targetNamespace for biztalk its not a problem at all and it is then guaranteed to work in ordinary ASMX as well.
(2) Serialization and Precompilation: The second problem was much more involved. Once we finished revising the namespaces, we called the webservice only to get a nasty error message saying "File not found exception. System cannot find the file "xyz123".dll (some arbitrary constantly changing name) . We tried setting all sorts of permissions everywhere and it didnt solve the problem. We narrowed it down to the serializer since ordinary webservice calls not involving complex types were working fine and since the webservice was writing to our debug log files we knew that the system wasnt even invoking the webservice. Finally we saw a post where someone recommended that we use Chris Sells' XmlPrecompiler tool and the associated GUI built by Matthew Nolton. We then used the WSCF tool to generate a bunch of classes corresponding to the schema and checked them out. By this time i had also narrowed down the problem to a particular set of webmethods and one in particular where returning a void worked fine but returning a strong type caused the crash. Now the tool worked perfectly and crashed at the offending type. On close examination we found that by mistake, someone had created an array of arrays. (ie) we were returning an element called References and in that we needed unbounded elements named Reference with some attributes. It turned out that References had also been declared as unbounded. To add to this the namespaces were a bit muddled up and i read an article on MSDN saying that unqualified attributes in array types cause problems (or something to that effect). Anyway, i basically rewrote the schema to ensure all the elements were qualified correctly, corrected the 'unbounded' element and then it worked fine. There were also other schemas that dealt with arrays where the namespaces had to be reworked, but in the end it all worked fine.
This is a problem that one doesnt mind spending a couple of late nights working through eh? Who would have thought it would come down to finding a precompilation tool and sorting out the bug. So there it is... be very careful with schema definitions and arrays and use the precompiler tool as part of your quality checks. It will save a lot of tears and gain you some sleep as well.
Whew.. this is a record for me. Two posts in the same day. Well, it has been a rather busy weekend trying to sort out my new laptop and make some progress on a serialisation issue i've been having. Thought i'd post about it cos it might make interesting reading for any Biztalkers in a similar situation out there. It also looks like this issue is still floating around on newsgroup posts (in fact I just came across my post on this subject last year and nothing much has happened since then).
[Update: Since this post, I've managed to solve the problem. So i've done an edit to put in the resolution]
Heres the situation
(1) You publish an orchestration as a webservice. The schema of the message is fully qualified
(2) You submit data via a . NET component. In the processing of serialising data you use the correct namespaces in the XmlSerializer.
(3) The orchestration recieves the data and throws an exception since the ns0: prefix is missing from the root element although it is there as a prefix for the main namespace (ie) it is there in xmlns:ns0="xyz".
A similar problem exists when you send data from inside Biztalk to an external component and find that the XML cannot be deserialised into the corresponding class because the parser crashes when it sees the ns0:.
So what do you do? Some people have a separate message for the webservice which has no targetNamespace at all and then manage to map it into a a more strongly qualified message in the orchestration. The problem i had last year was that when we opened the map again it threw all sorts of weird script errors and sometimes wouldnt compile. At that point we were sending data to an external system so we had to resort to a hack. We changed the component to accept an XmlDocument and then wrote another little helper to strip out the ns0 and sent the message on. This worked fine, but its a hack and i still find it distasteful.
Today Im in the situation where i need to submit data via a strong typed webservice to an untyped ASMX sitting on top of an orchestration.Sending the data to the strong typed facade is not a problem. But from there, I need to send it to the XmlDocument interface and just serialising the recieved object with a correct namespace doesnt solve the problem because we cannot prefix with ns0.
Resolution: To solve this we had to revisit the namespace declarations and ensure that all elements were prefixed correctly. Additionally, in Biztalk we had to make sure that the schemas were set to Elements - Qualified and Attributes- Unqualified. In a couple of our schemas the targetNamespace was redefined in another part with another prefix. All of this contributed to Biztalk not understanding the schema. When you send data to an external webservice, if you import the webreference, Biztalk creates the necessary schemas so all we need to do is map our internal message to the WS message and if there are prefixed elements in the schema definition, they will be prefixed in the transmitted message. If you are sending serialised data into an untyped message (Xmldocument input) or where the webservice takes a string as input (debatable practice, I know!!) , then the ns0: will go across in all the elements. However, if you use the same XSD on the other end, then you shouldnt have a problem in deserialising the data. In our old project, there was probably some foul up in the recipient schema which caused the serialisation problem. Its also worth experimenting more with XmlNameTables and suchlike to ensure the elements get mapped correctly.
Hope this helps someone in a similar situation.