BizTalk
All posts about BizTalk 2004 and Biztalk 2006
In my previous post i had outlined my first attempt at creating solution templates using the Visual Studio vstemplates technique. I have now gone one step further and built an add-in that allows the user to apply solution wide settings to all the projects. The settings that can be specified are as follows
- RootNamespace : Namespace prefix upto and not including the solution. This usually follows the format Company.Product. ProductGroup. (e.g) Microsoft.IntegrationPlatform.Services.
- Solution Name: Taken from the active solution, but can be changed to anything needed - for example CustomerService. The reason this is provided is because some teams like to keep the entire namespace in the solution name but while generating assemblies, the prefixes could be superfluous
- SolutionNamespace: Namespace comprising the rootnamespace and the solution. (e.g) Microsoft.IntegrationPlatform.Services.CustomerService where CustomerService is the functionality provided by the components of the solution.
- Application Name: This is the Biztalk application name to be applied to the biztalk projects. Sometimes this is the same as the solution name but need not necessarily be so.
- Key File: This allows the user to select an existing key file or to create a key file and have it applied to all the projects including the non biztalk projects (which is needed to allow them to be GACed).
Acknowledgements
The strong naming functionality is borrowed from the work done by Jon Flanders and Carlos Medina . In Jons tool you can generate and apply a strong name to a biztalk project on the fly and with Carlos' enhancements,you can generate and apply the strong name to all projects, or if you already have a strong named project you can copy those settings across to other projects. You can also apply the ApplicationName to the active configuration.
In this tool, the extra functionality is that you can
(1) apply the application Name to ALL configurations - deployment/development/debug/release etc (i usually add in the debug and release configurations to my biztalk projects to make it easy for my automated build scripts to move between biztalk and non-biztalk projects and
(2) choose a strong name file from anywhere on the machine and opt to use a hardcoded path to the key file or a relative path. At this time the relative path can only be calculated from the solution root so the program copies the snk file to the solution root and calculates it from there. Technically it is possible to calculate relative paths to anywhere in the source tree, but right now the code does not support this.
You can download the add-in from here.
As always, your feedback will be appreciated and welcome to help make this tool better. I am looking at trying out some wizards next which will be more flexible and allow more settings and options to be specified.
Just a quick note to say that Kevin has a beta release of BizUnit 2.3 on http://www.codeplex.com/bizunit. I had a look at the alpha version earlier and i must say that it looks pretty good. Theres a bunch of additions to the BizUnit() and Context() classes including the ability to pass a Stream to the BizUnit constructor which looks like a nice way to break the dependency on the raw XML. We now have the chance to use the Context in more depth including setting up property values into the Context and passing that into the BizUnit constructor.
I've queried about the documentation and whether CHMs will continue to be the only sources there. If so, then i plan to write more articles in the BizUnitExtensions project to cover the new stuff.
I havent had a chance to test extensions against this release, but there is definitely a breaking change in that the Context HasKey() and Change() methods that are used by some extension steps are not available anymore, but the good news is that the ContextManipulator step appears to work, so i guess i can change the code in the appropriate steps to use this step instead of Context Change, or i can move the context change and haskey() into the extension steps. Lets see how it turns out. I should be able to re-release an updated version of extensions as soon as 2.3 is complete.
Anyway, grab your copy of the latest BizUnit and test it out and post your feedback on the codeplex site. Enjoy...
I've always been fascinated with the concepts and technique of automating VS development through macros, add-ins, templates etc.By now at my clients site we have more or less standardised what our Biztalk "services" look like but since we are still in Biztalk 2004 I didnt want to waste time with VS2003 automation as things have become so much more polished and advanced in later versions. I recently spent a couple of days at another client site with v2006 setting up the solution tree and MSBuild scripts and then sat back and wondered why i didnt have a tool to set the whole thing up in 5 minutes. It would have saved so much time that could have been spent on more significant issues. So today, after much procrastination (and collecting links on the subject), I finally decided to take the plunge and create my own multi-project template.
There's a wealth of material on creating Visual Studio Item and Project Templates and one of the best I've read is How to create starter kits (including multi-project templates). I gleefully put in the .net class library and test projects, schemas, maps etc and ran the Export Template Wizard. Imagine my horror when i found that it would not pick up Biztalk projects!!. So i got stuck for a while. I did have a look at Jon Flanders awesome videos on his Pattern Wizards including the facility to create biztalk project wizards from your own projects. I say, that guy is really something!! Anyway, that helps to some extent in that we can now "export" existing biztalk projects and use them later (and make them highly dynamic through wizards etc), but it still doesnt help create a multi-project template.
So I went ahead and decided to try and experiment. With the starter-kits article in hand I created the basic .NET projects and then decided to take a set of biztalk projects and add them in. I removed all the unncessary items, except for the btproj file. I then added in a .vstemplate file and a icon file (from the exported .net projects). I then edited the vstemplate file to set the links to the btproj files and so on. Ultimately having put them all together and zipped it up and dropped it into the Templates folder, voila, it worked. I have a multi-project solution with Biztalk projects in it.
In the btproj files i can set macros such as $safeprojectname$.Unfortunately no matter what I try it will not accept items such as schema files, maps etc. VS always complains that it cannot find the files even if they are included in the package. So i gave up that route and left the projects empty. I would like to try to set the Application Name as the name of the Solution in all the btproj files, but there doesnt seem to be a template parameter to do that
You can find the results of my handiwork at my new codeplex project site here . Feel free to play with it and let me know your thoughts and suggestions.
I'm thinking of extending this with add-ins and macros to create msbuild scripts to build and deploy the solutions. This would give projects a good kick start. Of course, people modularise Biztalk in many ways, some have multiple schema projects, some have multiple orchestration projects and the layout is not always the same. And of course, once you start using Jons tool, then the solution will morph into something very different from what the starter kit looked. So the Build and Deploy script generator tools must be capable of running at anytime and regenerating stuff from scratch. The codeplex site should see some growth very soon. It would also be interesting to hear how others lay out their projects especially how branching is done.
I've also written up another little tool to generate my source tree, taking inspiration from tools like TreeSurgeon. I should put that up soon. For now , i guess these tools will work together, but in a slightly disjointed fashion and in time i'll get a deeper understanding of VS automation and with stuff like GAT, theres a lot that can be done.
The big limitation right now is that doesnt link in with Jons work in any way. All my project templates that i made are sitting in the Biztalk Project Wizards folder, but the vstemplate only shows its own collection of btproj files which arent dynamic in any way. If anyone has some experience on linking VStemplates with wizards and would like to sharet the info please let me know. I'd love to be able to link my templates to Jons wizards, thus making it a highly dynamic and rich experience in creating multiproject solutions.
Just to inform users of BizUnitExtensions, that Kevin Smith (creator of BizUnit) and I have been having some extensive discussions on the BizUnit codeplex site (here and here) on how these two projects will continue to work together going forward since BizUnit is now active once again and Kevin has lots of plans for it and a new release with lots of good stuff is imminent.
Once the new release of BizUnit comes out the Extensions project will no longer release the core as part of the distribution but will continue to release test step libraries as it is doing now (perhaps with some structural or namespace changes), so we will have our own release schedule. I'm hoping that the changes we made to the core to allow some of the extensions to work will be retained by Kevin and that other changes we are proposing such as making all the properties publicly accessible to support tooling plans will also be acceptable.
If you would like to influence these decisions please do weigh in on the discussions at the BizUnit site. I have had some good feedback about the extensions roadmap but do continue to send us messages on what would be the most useful features.
Update: I've now put up all the items on the roadmap as work items so why not hop over to codeplex and click on the VOTE links against each work item and let me know what your favorite proposed features are?
Martijn Hoogendoorn has re-posted the link to his utility to programmatically generate orchestrations (now that he is working for Microsoft the old link on his previous blog no longer works). Check it out
here. He has used the Microsoft.Biztalk.IMS.dll which contains interfaces such as IBtsVsDocument and the Biztalk Object Model dll to do most of the generation. This is quite different to Jon Flanders Pattern Wizards which have a wizard written over a bunch of VS2005 templates to generate what you need. In this codebase, Martijn has one class which generates a sample akin to the SDK CallOrchestration sample. Its pretty nifty and demonstrates how one can use the Biztalk TypeSystem to generate ports, message references etc. One thing which puzzled me is where he got the Biztalk.IMS.dll from!! Jon has that in his codebase too but i couldnt find that anywhere on my machine! Does anyone know where we can get that or where some documentation would be available?
[Update: Martijn informed me that the DLL is in the GAC and so it is!!]So when would you want to code gen orchestrations? After all, using the orchestration designer isnt exactly rocket science and now that there is a wealth of community wisdom in the realm of patterns (encapsulated in Jons tool), why do we need this? I look at this technique as facilitating one layer of abstraction over the designer itself. Take the "Solution Designer" that Eddie Churchill demo'ed in a channel9 video If you look at that, it shows how the Team Architect AD surface can be used to sketch your solution end to end and then generate the solution structure with stub orchestrations, schemas, maps etc. Thats a good application of DSL technology (although in the demo version it used windows forms, not the DSL toolkit which wasnt stable enough to be used at that time).
Anyway, back to the discussion. One thing about good architectures is that they should also be consistent (along with being flexible). So if you have a bunch of backend systems that expose similar interfaces to be integrated with (WS, queues, file ports etc) and if you have a choice of which interface to use, while you are well within your rights to use any way of connecting to them, IMO, a better way would be to establish a consistent pattern of connecting to them so that you can make sure your bases are all covered, your development team knows whats expected and you have a standard base for delivering estimates. So if you do this, then you can fairly well predict exactly what your "connector" looks like and then, if you have worked out the contract for your schemas etc, then you can just go ahead and code gen the entire lot. Use the VS automation model to generate the solution structure, add the various projects in programmatically and then inside each of them, use these utilities (like Martijns) to generate the rest (and in a future version of BizUnitExtensions, autogen your tests as well). This way we can maximize our "design contracts".
I do wish the Biztalk team would make the WebService Publishing Wizard available programmatically. (I guess the WSE publishing wizard is probably on its way out now that WCF is in production). This would help even more code generation. By the way, i heard that Eddie has left the BTS team and so the solution designer has been discarded. Anyone know the skinny on this? I would be terribly disappointed if this is the case because to me, that was the most compelling feature in the "future/vNext" Biztalk release (perhaps it was post R2).
By the way, I also came across another tool named BridgeWerx sometime ago that used a custom interface to sketch your biztalk solution and generate the codebase. I wonder if they have moved into the Microsoft DSL space now.
I've asked Martijn about the possibility of making this into a CodePlex project (although the source code is available with the download, putting it up as a project would make it more visible and get more collaboration). I'm sure that with more hands and more time (overall) there could be much more powerful things that could be done with the tool.
Hey everyone, I've finally managed to upload the v2006_R2 release. Its actually a rather small release and i know i've got to sort out my version numbering really soon. I promise to do that in the next release. I've been struggling with a VPC from hell that just crawls all the time and doesnt let me write any code at all and combined with the general lumbering behemoth that is VSTS, i'm just about stuffed Since i dont have anything installed on my base laptop(and the workstation in the office wont allow me to connect to CodePlex) i'm forced to work with the VM and i've spent long nights just disabling services and trying to get it to perform, but it wont. Ok, thats enough of my rant for the day.
This
new release has the first cut of entlib logging added in and a new
"MapExecutingStep" to test maps. The AltovaXmlValidateStep has also been
enhanced and it now allows the user to specify whether the file is expected to
fail or pass the validation (to cater to scenarios where we need to test invalid
files). I've also added in some strongnamed EntLib2.0 dlls and a snk file into the References folder. You can of course choose to use your own entlib dlls. There is no separate test as such for entlib logging and the same test base is run and the system logs to a logfile specified in the config file for the tests.exe. I'm thinking of making the core libraries as exe's because we can add references to exe's now and this way i can supply a set of entlib config files with the library if people are interested.
Check it out and let me know if you encounter any problems. If you open
the solution and the system complains about not being able to locate TFS just
let it remove all the bindings. When you run the tests the perfmon test still fails but thats because there are some hardcoded paths in the tests that i inherited and i still havent had time to fix them. But the step works so ignore the failing test.
The next version will incorporate the latest from Kevin Smiths core code base and i will make all the modifications needed to the BizUnit() and Context() classes, or i may just add another BizUnitEx() class and a ContextEx() class so i can continue making my changes without breaking any of his code. I'll also try and make all the properties public. From reading the description on his releases page and blog, it appears that Kevin is planning to do more stuff with BizUnit and im still waiting to hear from him about the possibility of collaborating. If i dont hear soon, then i guess it will be time to make a hard decision about whether to continue with the "extensions" tag for this project or just move to a completely separate code-base. Do stop by on the forums and let me know what you think about this.
Anyway, i hope you find the latest code base useful.
I know its been rather quiet on the BizUnitExtensions front, but never fear, theres lots of stuff to come. But firstly, i do hope Kevin Smith responds to my comments on his workspace on joining forces. Im still waiting patiently. Although its good to see a new BizUnit core release with bug fixes and changes, its actually beginning to get problematic now because we have made changes to the core of the BizUnit class and other classes such as Context and now we have to port everything to the latest BizUnit codebase in order to keep things consistent and doing this to a code base which has now structurally changed is not easy. Added to that is the fact that we also wanted to change the structure of the code base (so we keep the standard test steps in one, biztalk specific things in another, msmq in yet another etc) so thats got to match up as well. I suppose these are the kind of scenarios where the C# 3.0 extension methods would come in really handy. Maybe its a good time now to break away from the BizUnit label and move to the more generic integration testing framework. What do BizUnit (and Extensions) users think about this?
Anyway, a good discussion has started on the forums at the extensions site (www.codeplex.com/bizunitextensions) with one of the new users querying BizUnits actual value (Why BizUnit) and finding some answers in the roadmap. I guess the reply to the question would be worth a blog post on its own!
So whats cooking with the Extensions project? Well, first of all I need to put down a release calendar showing whats coming up (based on the roadmap) in the next few releases, but right now the target is to get all the step properties, constructors etc public and to get entlib logging working. I have finished the entlib logging and will make it available as soon as possible. Its rather raw right now (ie) its a simple log format that more or less mirrors the console log but to a text log sink. i hope that this will stimulate some discussion in the community so we can improve the kind of logging (to event logs, perf counters etc).
Making the properties etc completely public is proving to be rather involved and raises lots of questions. For instance, if one could execute a step (for example, filecopy) independently (by just setting the properties and calling Execute) and could do the same for all steps in the test case, then the question arises as to what value does the BizUnit testRunner provide ? Some possible answers are that it provides
(A) Execution context (but this could also be set and passed in) and the ability to keep a constant thread of logging through all the steps and
(b) also provides parallel execution (but MbUnit could provide that). (Speaking of MbUnit, it definitely looks much better than NUnit and i should try running BizUnit inside MbUnit to see what extra stuff i can squeeze with MbUnit.)
I think it might be a better option to provide the ability to set step properties in code but once we create a collection of steps, we would pass the collection to BizUnit and say RunTest(TestStepCollection). This way BizUnit can manipulate context and keep full control. The Execute() methods would all then become "internal" and available to friend assemblies rather than public. If you are using the tool currently or planning to then head down to the codeplex site and post your thoughts on this or email me.
Well, what do you know, looks like Kevin Smith has turned up again after a long hiatus. The BizUnit project has now been moved to CodePlex and i just pinged him to check if he wants to collaborate with the Extensions project as well. Hope something good comes of this. The extensions project has had a huge number of hits since the content and structure got revised and i'm still trying to squeeze some time from my schedule to write up the tutorials and all that, so if you've been waiting, dont give up, or better still, email me and let me know you're waiting. The last time someone did that, i wrote up the Context tutorial immediately so let me know if you want some information to be put on the site.
I should have taken Jon Fanceys advice (on his blog) seriously and "sacrificed chickens to the demo gods". I had to give a brief workshop on Biztalk to some new clients a couple of days ago and took the classic messaging & CBR example of mapping an employee message to an executive message. As with most demos, this simplest of scenarios failed miserably. Why are demos always like this? Being still stuck in the 2004 world for my current project(s) and demo-ing 2006 for the new project might have contributed,but its really the same product with a rather more shiny interface on top so i guess i had to chalk it down as one of those unexplainable things.
The problem we had was a weird one. Initially when designing the schema, one of the fields (Salary) which i promoted, was a string and then as i refined the schema i changed it to a "long". Unfortunately the property schema had already been set by then and that took me into a rather long loop of re-deploying, changing the prop schema etc since i needed to check the value of that field and set another element in the target (using a scripting functoid). The messaging error that popped up continually was "cannot convert string to long". We just couldnt understand it because we literally wiped the schema and started again and the system seemed to have cached that message and kept throwing the same error. Even disabling the send port which did the mapping and retaining another port which forwarded the incoming message to another folder didnt work. Anyway, ultimately we just gave up in disgust cos we were running out of time and there were other things to work through.
Still, i wont take "no" for an answer from Biztalk and i gotta probe this and get to the bottom of it. Its kinda embarassing to see things like this happen after you have been working with the product for quite a while, but hey, i've seen it happen to well known gurus in the community so why should i be an exception!!
Next time you are planning a demo, dont forget the chickens!!
Okay, so thanks to everyone who has downloaded the RC1 bits of the extensions library. It seems to be working fine and i havent recieved word of any problems (except for one small bug in the st