June 2005 - Posts
DAAB : The Ent Lib data block is still not behaving itself, so we've taken the soft option. Since the service we were working on is a web service, BTS doesnt really care that it uses Ent Lib, so we managed to hide the DAAB behind the web service so the two dont quarrel. For now we decided to let sleeping dogs lie.
Architectural Consistency Woes: The sad thing is that I'm struggling with getting the DAAB to insert NTEXT data into a database. It complains that SqlCommandWrapper cannot be used due to some protection level. See my GDN Message Forum post for details. Bah! Humbug! Now I've temporarily lost consistency in the architecture cos some services are using Ent Lib and the others are simple direct SQL Connections. Still, I need to get things done and I cant insist on consistency at the expense of getting things to work, can I? That would be too ivory tower-ish. Perhaps in a later iteration, when we've delivered some concrete stuff and can spend some time digging around, we will be able to find out what the problem is.
Deploying Ent Lib: On another, but related track I had some good feedback from Tom Hollander in the new GDN Ent Lib workspace where I asked if we could deploy code to production while retaining the MS namespaces (some people said that it had to be changed in the old App Block days) and how do we handle the issue of several components sharing Ent Lib DLLs when deploying.
But there is some satisfaction. Iteration-1 of my system is now 95% done. I can transform the source data, call the load service and archive the files (source and transformed versions) to a database at completion. We are now re-factoring the namespaces for all the projects and custom components to get some consistency into it. Future iterations will involve dynamic mapping, content enrichment patterns, possibly some de-batching and aggregators and so on. Will keep posting.
Hey, just to say that I will be at TechED Amsterdam next week, so if anyone would like to meet up do send me a message on the email there or through the delegate networking tool.
I'm really looking forward to TechEd. Last year was my first and i didnt know what to expect. This time I think I know how to get the most from it. I wont be needing to go near the Hands On Labs at all since I'm already doing BizTalk hands on and I have VSTS installed on a workstation (not that I've actively used it since installing, but thats another matter).
The EventCalendar and the nice print out it gives makes it easy to see where I've double booked. Last year it was quite tough since i didnt know how to use those facilities and ended up working on a massive spreadsheet to divide up session attendance between my colleague and myself. The annoying thing is that as of today only 5 PPTs are available so I cant make up my mind about sessions where I'vr double booked.
I have the downloaded webcast of Ron Jacobs talking about Patterns for SOA, so i might not actually go for that, but the slides are really interesting, so I might end up changing my mind.
Solving the Web Service communication issue: Thanks to a great post from Matt Meleski on passing complex messages to web services, we sorted the problem with communicating between the Orch and the WS. Basically we were very close. I sort of knew we werent doing the correct thing by assigning the WS request message directly with the Target Data message and that we should be dealing with variable assignation instead. As it turns out, I was right. We first have to load the incoming message (or the transformed one, whichever you want) into an XmlDocument variable. Then if you are using the XmlDocument as a parameter in the webservice, you can simply assign the web service parameter to the .OuterXml of the variable. Matt gives more examples of using strings as web service arguments as well as complex types as arguments.
But cannot handle EnterpriseLibrary.Data: Looking through Andy Morrisons blog on using Ent Lib Logging Block was fine and useful, but it does not address the beast that happens to be lurking in the new DAAB. The event log always says that it failed to load the assembly even though its clearly marked in the BTSNTSVC.exe.config. Actually the problem is in the dataConfiguration.config file since the BTSNTSVC file points to this and it would appear that BTS is able to go to the dataConfiguration file but just cannot read the config section properly. The funny thing is that on its own, the new DAAB and the web service work fine (when tested using NUnit) but when called from BizTalk they start acting funny.
Weird huh? We're going to give it just one more shot tomorrow for a couple of hours and then we'll scrap Ent Lib completely. Will keep you folks posted.
The requirement: Convert schemas of type A1, A2 etc (various diff forms) into a common schema, B and then load the message into a custom database.
In my previous post I explained that we finally got the data load component working. Next we were asked to re-factor it so that the loading could be done through a web service. That started another round of hair loss. We created a web service interface for the data loader and created a method called ImportData taking a class corresponding to schema B (generated by XSD.exe). Now when we added the web reference, it created a WebService_Request message and a WebService_response message.
In the transform, we had already created schema B, so one would think its easy to pass it to the WS since after all they are the same thing arent they?. Thats where we fell down. We put in a message assignment shape to assign WS_Request to the Target_Msg created earlier, but it wouldnt let us assign or cast it. We couldnt simply send the WS_request because, obviously it hadnt been assigned. In fact it wouldnt even compile and said that the request message was not initialised and couldnt be used.
To work around this we thought we could construct a new message of type WS_Request and at least MAP the Target_Message to WS_Request (although the mapping is redundant since the schemas are identical). We created the map, but in the Transform shape where we indicate the map and the source and destination, the destination drop down simply refused to pick up the WS_Request. I wonder if it is a namespace issue.
So that left us stymied. Without the web service life is difficult because we are using Enterprise Library in the data loader and all attempts to set up Ent Lib to work with BizTalk (including using BTSNTSVC.exe.config) have failed. The system always complains that it cannot find the needed configuration section. At least with the web service we can use the web.config so that Ent Lib does not complain.
Anyway, the plan now is to (a) get rid of enterprise library and either use the old DAAB or write our own and (b) to remove the redundant Target_Message mapping and try to construct a message of type WS_Request directly (assuming it will appear in the Transform drop down). Might also look at the PUBLISH SCHEMA AS WEB SERVICE option to see if that helps.
Does anyone have any ideas on why the ws_request doesnt appear in the Transform drop down. Would appreciate any feedback.
We finally started development with BTS and as expected, the first two days were rife with problems.
In our system we are getting a load of data files in different formats that need to be converted into a common schema and then loaded into a database. So we designed the common schema and started on a little data pump which takes the messages, deserialises them and loads them into the database. We didnt try the SQL adapter and updategram route, but we will put a spike in at Iteration 2 and look at that.
Day 1: To start off, we just wanted to see how to call a .NET component from an orchestration and this started off a string of problems. Firstly all the basic samples in the tutorials just dealing with transforming and copying files stopped working. All that HAT said was "Unparsed Interchange" which apparently means that the received file is not recognised (and this for samples that have worked previously in all my demos). For one whole day it didnt work. I posted this on the MSN BizTalkServerStuff group too but to no avail. Finally at the end of the day, just when i was packing up and leaving, one file actually got copied to the correct destination!! Why? no idea. It just started working.
Day 2: Now we had to call a .NET component which takes the data and just dumps it into a file (just to see if we could call our own components). That was yet another struggle. It took a combination of an article by Darren Jefford and another two by Mike Holdorf (parts 1 and 2) + some brainwaves and finally it worked. In the middle of that it took a fair amount of sifting through the Bloggers guide and the BTS Unleashed book to find out how to get debugging working and how to step into VS.NET code (for your component) while debugging an orchestration. The latter didnt work at all inspite of assidously following instructions but we managed to get the basic debugging and stepping from one shape to another working.
Here's the deal on calling your own components
(1) Create a component (strong named) that has a public method taking in either a XmlDocument or an XLANGMessage. Mike H's articles (linked above) will explain the namespaces needed etc. If you are using XLANGMessage you will need to use a RetrieveAs() method on the XLANGMessage object to get back an XmlDocument or Stream or whatever you need. Note that if you use XLANGMessage then you cant actually unit test the component cos you cannot create XLANGMessages outside BizTalk. XmlDocuments might be a better option if you want the unit tests to work.
(2) Include a reference to the project with the component in the orch project.
(3) Create a new orch variable of type 'your component'
(4) In the orchestration, after the Construct Message is finished, drag an EXPRESSION shape in the same path/branch. Ensure that the expression shape is NOT in the scope of the construct message. If it is within the scope it wont call your component.
(5) Edit the expression. use something like
myObject = new <Namespace>.<Class()>
myObject.DoSomething(incomingMessageInstance)
(6)Ensure that the user for the BTS Host Application (running the orchestration) has permissions to do whatever is needed on the folders. In my sample it dumped the data from the incomingMessageInstance into an XML file. In the official code it will de-serialise into a set of business objects.
This worked for us. I didnt have to debug after putting in the expression shape, although i suspect that my earlier problems with stepping into my component code were because it was within the scope of the construct message.
Hope this helps y'all.
More to follow.
Well, I just found this post on the SYSCON site saying "As announced by Paul Flessner during his keynote on Tuesday, BizTalk Server 2006 will launch on November 7th as part of a major three-way launch of BizTalk Server 2006, Visual Studio 2005 and SQL Server 2005. While the launch is planned for November 2005, the actual release to manufacturing of BizTalk Server 2006 will happen in the first half of 2006"
So I guess that clears it up, sort of. Its a mix of a product release and a launch announcement. Since there will be CTP's released at TechEd USA and Europe this year, perhaps there will be a first beta? Im eager to see what they are going to do for deployment, especially when you consider the amount of pain Martin Hellspong seems to have had and my comments on the same. If you notice, in the reply to that post all they said was he had better attend TechEd 2006.
As I indicated in the previous post, I am finally getting my hands dirty with ol BizTalk 2004, so i should be able to see if I run into the same problems as Martin Hellspong.
I finally started a nice project in BizTalk 2004. [Unfortunately I cant reveal more about it, but you'll have to take my word that its going to be really cool - as if you cared :-)].
We rolled up our sleeves, fired up the XML Schema Editor and ...... crash landed into a bug (or some odd behavior.. maybe it was just us). The thing is, while creating schema B, we could not create a Child Record and assign it to Schema A, although the namespaces were the same for both A and B. Funnily enough in the recently completed training course, there was a similiar exercise which worked perfectly. I looked around for articles on the net and all the posts about this took it for granted that it just worked (i.e) - click on the element, go to the properties window and choose the required schema from the Data Structure Type drop down. Unfortunately my Data Structure Type drop down only shows xs:any and the current schema under construction.
Anyway, after some lost hair, we switched to XML Spy 2005, created the schema there and imported it into BizTalk. We also found that Spy 2005 has the ability to create BizTalk schemas (given a sample XML file) directly, but since we didnt have a sample file available, we just hand crafted the schema.
I dont know what to make of it. I had just finished telling a colleague (and team mate) about the 'glories of BizTalk and its 'real nice' schema editor and this happened. He found it rather funny.
Let's see.I'll try again next week when I have some time.
Scott Woodgate has emphasised in earlier posts that the 2006 in "BizTalk 2006" clearly means it will be released in 2006. So why is MS muddying the waters with this press release, saying that it will be in Nov 2005 along with Whidbey and Yukon?
I suppose they mean information about BTS 2006 will be made available at the time of this launch (for those who arent upto speed on whats happening in the BizTalk world) but the wording is rather confusing.
I got a flood of nearly 50 spam messages in a single day (a couple of days ago) and Donny Mack has said that they definitely are doing something ASAP. So i'm waiting.
Anyway, on the lines of my previous post on Whitehorse extensibility, here's my postulation on extending the class designer. My idea is to use the class designer and with OR mapping principles, make it generate a database design.
The generation will have to highly customizable because DDL generation is a highly sensitive thing for database folk and from personal experience i know that some OR mappers (that take class design as input) generate some primitive DDL thats quite useless.
It should be interesting to take this further into XML message design and so on.
I just found a really cool blog with posts from the Whitehorse team. There are articles on how to generate the Deployment Report, manage the dozens of constraints and settings in the Application Designer and so on.
I think there is great potential in the Whitehorse toolset. Here are my thoughts/visions on the extensibility. Some of these might be basic (and folk might be doing it already) and some might be impossible, but hey, lets see.
(1) Extend it so that it generates projects using custom enterprise templates.
(2) Add in symbols for Ent Lib so we can add in the required blocks automatically
(3) A 'deployment package' view where you can state which package hold which assembly and where it goes. For instance, in my previous project we had a mix of console apps, DLL's and web apps. When we deployed them we combined some console apps into one project and used command line switches to execute the required pieces. We could potentially do this diagrammatically.
(4)Taking this one step further, you could probably link into App Centre when deploying to farms. I would also like it to trigger the entire build and deploy process. Jeff mentioned (in the above said blog) how some people were just happy with the XML deployment report and wanted to script their own deployments cos they didnt want MS putting stuff on their boxes!! Thats fine for some. I'd be happy to have an automated deployment option.
There's some of my ideas. Does it resound with anyone? think its rubbish?
So we finally got rid of VPC completely because of the problems with Yukon and rebuilt the box with Windows 2003. Now everything connects properly. We've started a couple of sample applications to run through the features.
Since there are two developers on this work we've split it such that I check out Team Architect and my colleague checks out Team Developer. We'll then try out stuff like MSBuild and see how that goes.
I'm still around. The folks at CodeBetter dont seem to want to give me a blog, so I'm going to stick around here for a bit. Donny has disabled some file that deals with comments so the spam has reduced now (only 2 yesterday - or maybe that was before he disabled it.)
So I finally hit the Whidbey and Yukon trail and am stumbling a bit now. I'm using the DVD set i got at the DeveloperDeveloperDeveloper event.
Using the install guide I managed to get the base products installed. My config is : DELL workstation (1 GB RAM) running XP and Virtual PC 2004 running Windows Server 2003. TFS is on the VPC and the VS 2005 tool and the client are on the main workstation.
The problems are
(1) There is not enough documentation (or not enough GOOD documentation , more accurately) on how to get started. The Install Guide is just that, an install Guide. After that you have to connect to Team Foundation Server (if you want the team stuff), but it does not tell you how to handle the errors and problems there. Luckily i found this post that deals with connection problems to the TFS. My problem was that my VPC 2004 instance (running the servers) was logged in with domain user X and the parent workstation which had the client was logged in with domain user Y. Both were admins on the machines, but that didnt help.
(2) The SQL Server Management Studio is corrupted. It gives some horrible interface registration error messages and refuses to launch.
(3) AdventureWorks sample database doesnt get installed, even when explicitly chosen. By the way, you have to go into the ADVANCED option and into the Books/documentation option to get at the sample databases. So beautifully intuitive (not). Apparently it just puts an MSI into the Tools/Samples folder and then you have to run the MSI.
(4) You cannot REPAIR an installation. I dont remember if we could do this in SQL 2K but this time its really bad. I wanted to reinstall Management Studio but the install application wanted to reinstall the lot. These things have a mind of their own. Unfortunately for me, im rather stuck now since I cannot reinstall without overwriting the TFS stuff.
(5) Not enough RAM. This is not Microsofts problem. They recommend a machine with 1 GB RAM for the server. Added to this, VPC is slow so the install , esp SQL, just crawls like a dying animal.[I tried installing SQL on the base machine and it went quite quick] I can only give 600MB to the VPC. Generally this VPC thing is beginning to look very dodgy indeed. Im going to need 4GB of RAM on the workstation if both are going to be running simultaneously.
(6) How do you test the installation? Nothing around to tell you, try out so and so application which will exercise the suite completely so you can know its fine. In an era of TDD is an installation sanity checking app too much to ask?
As a side note, I got a bit of a fright when I tried to create a web service and couldnt find the project template to do that. I finally found it under the 'Web Applications' option. Whew!!
Has anyone come across problems with the Yukon April 2005 CTP? Drop me an email through the Contact link and let me know.
Anyway, I got my first assignment on this. I have to do a presentation comparing and contrasting MSBuild with NAnt/CC.NET.