Friday, September 30

BizTalk ESB Toolkit 2.1 samples (Content based routing)

I finally came round to a blog I'd been thinking of doing for some time now, since I've been recently hit by the blogging bug...

Sometime ago, Abhijit Mahato posted this great blog about the ESB toolkit 2.0. and Visual Studio 2008. I stumbled acros this post once and was impressed by how simple it was. I therefore thought it would be good to simplify it further one day and add it to my jargon-free IT blogs whenever I finally got round to it. Well, the day has arrived

I have now re-written that blog (in a lot of the post, I have used his exact paraphrasing, so all credits to him. but I have also re-arranged some of the steps to make them more logical and more standards-friendly) with extra notes for the ESB toolkit 2.1 implementation with BizTalk Server 2010 and Visual Studio 2010. I would advise downloading Abhijit's source files so you don't have to re-create the schema and test files. Hope someone finds this useful

In this post, I am going to demonstrate how to achieve content based routing using BizTalk ESB Toolkit 2.1 and the BTS 2010 Business Rules Engine, using a business rules policy.

The Business Case:
Employee data is to be picked up at a file receive location and automatically routed at runtime to different destinations based on the departments of the employees.

Walkthrough:
In the walkthrough, the business rule policy (RouteBasedOnDepartment) is used to set the endpoint information of the incoming employee messages based on the value of the department field (node) in the incoming XML message.

The Itinerary (RouteBasedOnDept.Itinerary) contains an ON-Ramp (ESB Receive Port), OFF-RAMP (ESB Send Port) and Routing service with a BRE (Business Rules Engine) resolver to dynamically route the message. A business rules engine is a software system that executes one or more rules specified by a business in a runtime production environment. See this link for information on the microsoft BRE.

Create the BizTalk Application
Open Visual Studio
Create a new solution and call it ESBSample. Call the Project ESBSample.Schemas

Add the Schema
Right-Click on the Project ESBSample.Schemas and Add New Folder
Name the folder EmployeeSchemas
Right-click on the employeeSchemas folder and Add Existing Item. Browse to the file employee.xsd in the location you downloaded Abhijit's source files to and click Add


Add Deployment Settings
Right-Click on the the ESBSample.Schemas project and click on Properties
Click on the Deployment tab
Set the Application Name to ESBSamples   (note the s at the end)
Click on the signing tab. tick the Sign the assembly box. In the 'Choose a strong name key file' box, Click on <New..>
Type in ESBS.snk (or any other name really) into the 'Key file name' field
Untick the 'Protect my key file with a password' box
Click on OK
Deploy Application into BizTalk
Right-Click on Solution 'ESBSample' at the top of the Solution Explorer
Click on 'Deploy Solution'
This should deploy your applicaiton to BizTalk

Set Application bindings  and create folders
Open BizTalk Administration Console (typically start --> All Programs --> Microsoft BizTalk Server 2010 --> BizTalk Srver Administration
Right-Click on BizTalk Group[Your Server Name:Your BTS Management DB Name] at the top and Refrsh
You should see the application ESBSamples listed under the Applications node

Create File Receive Port and Location and configure
Right-click on Receive Ports, click New --> One-Way Receive Port
Configure it as follows:
Under the General tab:
Name: ESBSample.OnRamp.Itinerary

Under the Receive Locations tab:
click on 'New...'
Name: ESBSample.OnRamp.Itinerary.FILE
Type: FILE --> Click on configure
Receive folder: Click on Browse. Browse to C:
Click on 'Make New Folder'
Call it BizTalkESBSample. Make another new folder in this folder
Call it Input
Click OK 2ce
Receive Pipeline: ItinerarySelectReceive. Click on the button next to this field
Configure it as follows:
ItineraryFactKey = Resolver.Itinerary
ResolverConnectionString = ITINERARY:\\name=RouteBasedOnDept;version=1.0
RouteBasedOnDept is the name of the Itinerary which we will be creating later on


Create Dynamic Send Port and configure
Right-click on Send Ports, click New --> Dynamic One-Way Send Port
Configure it as follows:
Under the General tab:
Name: DynamicSendToOutput
Send pipeline: PassThruTransmit

Under the Filters tab:
Click on 'Property', scroll to Microsoft.Practices.ESB.Itineraries.Schemas.IsRequestResponse
Click on 'Value', type in False

Add the other following filters:

Microsoft.Practices.ESB.Itineraries.Schemas.ServiceName == ESBSamples  And
Microsoft.Practices.ESB.Itineraries.Schemas.ServiceType == Messaging And
Microsoft.Practices.ESB.Itineraries.Schemas.ServiceState == Pending And

Click OK

An explanation of the last part:
We have created a send port which will dynamically decide where to route the employee data message based on whichever message properties it finds promoted in the message context (The message context is a set of added properties which are not a part of the xml message body you can see normally. They are promoted - which simply means they are made viewable to the Biztalk engine - so that the Biztalk engine has instructions with which to perform functions and processes on the message.

When the Send port is created, BizTalk generates a set of subcriptions based on the filters specified
-----------------------------(you can see these subscriptions this way:
Click on BizTalk Group[Your Server Name:Your BTS Management DB Name]
Click on New Query
In 'Value' (Next to Search For... Equals) select Subscriptions. you will see all current subscriptions including the ones you have just created. the 'Name'  column will be the name of your dynamic Send Port.
----------------------------

However, an interesting thing to note is that, because this is a dynamic send port, BizTalk has no idea what kind of TransportType we will be using on this port (that information will come in the context of the message at runtime as explained earlier). Biztalk therefore generates a set of subscriptions for each transport type. To make sure we don't end up with the same exact subscriptions for multiple transport types when the routing informaation is being processed, BizTalk intelligently differentiates the subscriptions by adding it's own OutBoundTransportType filter (for each type possible) with a new AND logic to the filter expresssions we created. you can view this by double-clicking on one of the subscriptions and clicking on the expression tab

Create Send Folders
Create a folder called OutPut in C:\BizTalkESBSample
Create Folders Admin, HR and Others in the OutPut folder

Create the business rules policy
Open Business Rules Composer  (typically start --> All Programs --> Microsoft BizTalk Server 2010 --> Business Rules Composer)
Create Policy
Under Policy Explorer, Right-click on Policies, Add New Policy, Name it 'RouteBasedOnDept'
Right-click on 'Version 1.0 (not saved)' and Add New Rule. Name the rule 'ForAdmin'
In Facts Explorer, click the XML Schemas tab, right-click Schemas, and then click Browse.
Browse to the Employee.xsd file in your ESBSample.Schemas project and open it.
In the Rule window, right-click Conditions, point to Predicates, and then click Equal.
From Facts Explorer, drag the Department element to the argument1 node under Conditions.
Click the argument2 node, and then type Admin.
From Facts Explorer, drag the Set End Point Outbound Transport Location definition under the vocabulary ESB.EndPointInfo to Actions.
Click and then type C:\BizTalkESBSample\OutPut\Admin\%MessageID%.xml.
From Facts Explorer, drag the Set End Point Outbound Transport Type definition to Actions.
From Facts Explorer, drag the Adaptor Providers definition under ESB.TransportTypes, vocabulary to <empty string>.
In the Actions pane, expand the Adaptor Providers drop-down list, and then click FILE


Similarly add two more rules named 'ForHR' and 'ForOthers' having following Conditions and Actions respectively.
      Rule: ForHR
      Conditions:      Abhijit.ESBSamples.Demo1.Schemas.Employee:/Employee/Department is equal to HR
      Actions:      Set End Point Outbound Transport Location to C:\BizTalkESBSample\OutPut\HR\%MessageID%.xml      
Set End Point Outbound Transport Type to FILE
    Rule:ForOthers
      Conditions:     Abhijit.ESBSamples.Demo1.Schemas.Employee:/Employee/Department is not equal to HR
                                                                       AND
    Abhijit.ESBSamples.Demo1.Schemas.Employee:/Employee/Department is not equal to Admin
   Actions:   Set End Point Outbound Transport Location to C:\BizTalkESBSample\OutPut\Others\%MessageID%.xml
   Set End Point Outbound Transport Type to FILE

Publish and Deploy the policy to RuleEngine database. (make sure to deploy the policy): Right-Click on 'Version 1.0 (not saved)'. Click on Save. right-Click again, click on Publlish, Right-click again, click on Deploy

Create and Deploy Itinerary
Now switch to the Visual studio solution and right click on the Solution, Add new project.
Choose a C# Class library and name it 'ESBSample.Itinerary'
Right-Click on the 'ESBSample.Itinerary' project and add a new BizTalk ESB Itinerary Designer named 'RouteBasedOnDept.Itinerary'

Configure On-Ramp


From the Toolbox, drag an On-Ramp shape onto the designer surface.
Name it 'RcvEmployee'
For Extender (in the Properties Window) select On-Ramp Extender
For BizTalk Applicaiton, Select ESBSamples
for Receive Port, select ESBSample.OnRamp.Itinerary

Configure Off-Ramp



From the Toolbox, drag an Off-Ramp shape onto the designer surface.
Name it 'SndEmployeeInfo'
For Extender (in the Properties Window) select Off-Ramp ESB Extender
For BizTalk Application, Select ESBSamples
For Send Port, select DynamicSendToOutput

 Add Itinerary Service


From the Toolbox, drag an Itinerary Service shape onto the designer surface in between the On-Ramp and the Off-Ramp. This is the service that will represent the processing action that occurs to a message between the two.
Name it 'SetDestinationInfo'
For Itinerary Service Extender (in the Properties Window) select Off-Ramp Extender
For Off-Ramp, Select SndEmployeeInfo, open the (+) node, and select Send Handlers

Configure The Actual Resolver Service



Right-Click in the white space in the middle of the Itinerary Service shape
Add New Resolver
Name it 'RouteEmpDetails-BRE'
For Resolver Implementation, select BRE Resolver Extension
For Policy select RouteBasedOnDept
Make sure UseMsg and Recognise Message Format are set to True

Deploy Itinerary
Click on the Itinerary designer surface
Set Itineray Status to Deployed
Set Require Encryption Certificate to False
Set Model Exporter to Exporter | Database Itinerary Exporter
Make sure BizTalk Server Connection String and ItineraryDatabase are pointing to your Database Sql Server instance
Right-Click on the Itinerary Designer surface and Click on Export Model (this should export your itinerary to SQL server.
If you would like to know what this means, open sql Server Management Studio and connect to your instance of SQL Server
Open Databases
Open EsbItineraryDb
Open Tables
Right-Click on dbo.Itinerary (this is the table in which itineraries are stored). Click Edit Top 200 Rows
You should be able to see your itinerary in the table. the version is stored in stored in the nMajor and nMinor fields, the actual XML configuration information (itineraries are simply xml configuration isntructions which are made simpler to create by the itinerary designer interface) is stored in the imITML field

Test the Application
From the BizTalk administrator console under the application “ESBSamples” enable the Receive location and start the send port you created at the beginning of the walkthrough.
Browse to the samples folder in your downloaded copy of Abhijit's source files
Now drop all the three files from the folder '\Abhijit\Abhijit.ESBSamples\Sample\' into  the 'C:\ESBSamples\Input\' folder. If everything is configured properly (fingers crossed!) then you will see one file each in the locations C:\BizTalkESBSample\OutPut\Admin, C:\BizTalkESBSample\OutPut\HR and C:\BizTalkESBSample\OutPut\Others, routed based on the value of the Department field of the input messages.

S

Thursday, September 29

BizTalk Development Errors: simple jargon-free problem and solution approach

P: You are using xml.xmldocument for a soap message type (in order to access it's properties for use in the ESB portal)  whilst implementing exception-catching in BizTalk orchestration, after assigning your message type to your exception object name, you get the error: "cannot implicitly convert type 'mesage global_NS' to message system.xml.xmldocument":
S: You have probably overlooked your exception object type. Make sure that your exception object type is properly derived from an xml message type - if it is derived from a port fault message, make sure it has a message type assigned to it)

P: sqlservr.exe is using too much memory resources and affecting deployment of your biztalk applications due to out of memory errors
S: Fix SQL Server’s memory allocation. Go to SQL Server Management Studio, Right-Click on the Server, Properties à Memory. Set the maximum server memory (in MB) filed to a value your system can deal with, like 200MB for example

P: you get an ‘errors exist for one or more children’ error message when you’re trying to build a BizTalk orchestration project
S: this would usually be due to a problem in the scripts within an expression shape (the problematic one is usually identifiable by the red exclamation mark), or simply look through all your objects in the orchestration window to find any objects with the red exclamation mark. – This post http://stevestechnotes.blogspot.com/2010/07/biztalk-errors-exist-for-one-or-more.html also suggests some solutions

P: You get persistent ‘Schema referenced by Map "*****" has been deleted. The local, cached version of the BizTalk Server group configuration is out of date. You must refresh the BizTalk Server group configuration’ errors. Your list of BizTalk Applications isn’t showing in the Group. You try refreshing the BizTalk Group and it doesn’t solve the problem. You also try restarting host instance services and the [problem persists
S: See first part of the next solution below

P: When trying to use the baselangs.Any schema type for a map, you get 'Schema referenced by Map <mapname> has been deleted' error. Your BizTalk applications stop loading up in the Administration console.
S: If you’re only interested in getting your applications back in Administration console, open SQLserver, open the BizTalkMgmtDb, and open the dbo.bt_MapSepc table. Find the entry whose input (indoc_docspec_name) and output (outdoc_docspec_name)  schemas match your problem map and delete if from the table. This will remove the problematic map and allow the applications to load in BizTalk Administration Console.

However, if you’re hell bent on using your map and you wish to fix the problem, follow these steps. Delete your problematic application. Refresh the BizTalk Group. Add the Microsoft.XLANGS.BaseTypes dll to your BizTalk Group by adding it as an assembly to BizTalk Application 1. (Richt-click on BizTalk Application 1 à Add à BizTalk Assemblies à Browse to your BTS installation folder and pick the dll. Click on OK. Re-deploy your application

P: When trying to deploy you BizTalk solution, you get error: Parameter name: StrDBName:
S: The problem would usually have to do with the deployment properties in the btproj file (Properties -> Deployment -> Server) trying to deploy to a server to which you do not have access. It is probably set to another server other than your local sql server instance. Either change the server name to your local sql server instance, or if you are trying to deploy to a remote server, make sure the domain name is added to the server name (e.g. Sql server Devbbl02 on Domain btsdev.dev would be entered as Devbbl02.btsdev.dev)

P: when an xml message passes through a port, you get error: "An error occurred when parsing the incoming document: "Invalid Character in the given encoding"
S: The problem would usually have to do with improper or absent declaration in the xml file. BizTalk uses UTF-8 encoding by default in the disassemble and assemble stages of the receive and send pipelines. If as an example, a file in ANSI format containing foreign charcters (basically non-English characters - excuse my partialism) is passed through the receive port using the XMLReceive pipeline without the declaration (i.e. for the default U.S ANSI charset it would be "<?xml version="1.0" encoding="windows-1252"?> the pipeline simply would not be sure how to convert the byte representation of the foreign charcters (since different encodings use different byte representations for foreign characters). the pipeline would therefore not allow the xml file through and the error above would be thrown.

The easiest solution (which of course would almost never happen in the integration world) would be to have the declaration added to the xml file form source

Another solution is to use a custom pipeline component such as the one on this link. You would add it to the decode stage of the receive pipeline to let bizTalk know what encoding it will be dealing with. You then use that custom pipeline in the receive location

Finally, you could write a custom pipeline component to convert from ANSI to UTF-8 and use it in a similar way to the custom pipeline described above

ESB errors: simple jargon-free problem and solution approach

The BizTalk ESB toolkit is a pleasure to use when it works just right, however, getting it to that point is one of the biggest headaches due to it's interconneciton of artifacts and rules... I just thought I'd make a growing journal of errors I encountered in my ESB implementations over the years. Hope these will help someone out there at some point...

P: You get Http 503. Service unavailable error when you try to open the ESB portal
S: Your web server is experiencing issues. Most likely the service is down or your apppools are not properly running. Check your Application Pools, make sure the Application Pool for ESB is started (usually ESBPortalNetworkAppPool). Open command prompt and restart IIs by typing iisreset <Enter>. Then try browsing to the ESB Portal again

P: When trying to access the ESB home page http://localhost/esb.portal/ you get an error saying there is an unhandled exception. In the error log the exception looks something like this:

Event code: 3008
Event message: A configuration error has occurred.
Event time: 22/02/2011 08:14:13
Event time (UTC): 22/02/2011 08:14:13
Event ID: 6e985cae3c3349a091823d65edf78b6a
Event sequence: 1
Event occurrence: 1
Event detail code: 0

Application information:
    Application domain: /LM/W3SVC/1/ROOT/ESB.BizTalkOperationsService-8-129428360532405064
    Trust level: Full
    Application Virtual Path: /ESB.BizTalkOperationsService
    Application Path: C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Web\ESB.BizTalkOperationsService\
    Machine name: DEVBIZ102

Process information:
    Process ID: 24196
    Process name: w3wp.exe
    Account name: NT AUTHORITY\NETWORK SERVICE

Exception information:
    Exception type: ConfigurationErrorsException
    Exception message: Could not load type 'System.ServiceModel.Activation.HttpModule' from assembly 'System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
   at System.Web.Configuration.ConfigUtil.GetType(String typeName, String propertyName, ConfigurationElement configElement, XmlNode node, Boolean checkAptcaBit, Boolean ignoreCase)
   at System.Web.Configuration.ConfigUtil.GetType(String typeName, String propertyName, ConfigurationElement configElement, Boolean checkAptcaBit)
   at System.Web.Configuration.Common.ModulesEntry.SecureGetType(String typeName, String propertyName, ConfigurationElement configElement)
   at System.Web.Configuration.Common.ModulesEntry..ctor(String name, String typeName, String propertyName, ConfigurationElement configElement)
   at System.Web.HttpApplication.BuildIntegratedModuleCollection(List`1 moduleList)
   at System.Web.HttpApplication.GetModuleCollection(IntPtr appContex

S: Run this command at the command prompt

C:\Windows\Microsoft.NET\Framework64\v4.0.30319>aspnet_regiis.exe –iru

If this still haent resolved the issue, make sure to re-allocate the ESBPortalNetworlAppPool is still using the v2.0 .Net Framework (Open IIS, Click on Application Pools  -àESBPortalNetworkAppPool à Right-Click àAdvanced Settings à .Net FrameWork Version
P: When trying to use an Itinerary published using the Database Exporter Model in BizTalk ESB, you get the error: “The Itinerary “<ItineraryName> was not found in the repository”
S: Two possible solutions
a)      You forgot to set the Fact Key in the On-Ramp (Receive Port) Receive Location’s Receive Pipeline. Open up the Pipeline properties and set the Itinerary FactKey to Resolver.Itinerary
b)      You actually forgot to export your Itinerary (d-uh)
c)       You forgot to set your itinerary to deploy before exporting if from the itinerary designer. Go to SQLServer (using SQLServer Management Studio), open Databases à ESBItineraryDb à dbo.Itinerary à Edit Top 200 Rows. Find the row corresponding to your Itinerary (where the column strName is the name of the Itinerary). Check if the column nStatus of is 0 (it needs to be 1). If it is 0, go back to Itinerary designer, open the Itinerary and click on the Itinerary designer surface. In the Properties Window, set Itinerary Status to Deployed and export the Itinerary again

P: When exporting your Itinerary to the Database, you get the error 'A X509 Certificate is required in the model property ‘EncryptionCertificate’ to encrypt any sensitive property in the designer'

S:
a) For BizTalk 2006/2009 with Toolkit 2.0 - In the Registry Editor, navigate to the subkey HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk ESB Toolkit\2.0\Designer, and then set the RequireX509Certificate property value to false. (If you're using a 64-bit machine, use HKEY_LOCAL_MACHINE\SOFTWARE\Syswow64\Microsoft\BizTalk ESB Toolkit\2.0\Designer)

b) For BizTalk 2010 with Toolkit 2.1 - You actually only need to set the Require Encryption Certificate value in the Itinerary Properties to false. If you also wish to change the registry keys, in the Registry Editor, navigate to the subkey HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk ESB Toolkit\2.1\Designer, and then set the RequireX509Certificate property value to false. (If you're using a 64-bit machine, use HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\BizTalk ESB Toolkit\2.1\Designer)