BizTalk 2010 message submission with transport-level security to SharePoint 2010 forms library without the WSS adapter

This was a solution I designed and built on one of my projects. The idea here was that the client had a process which needed to receive foreign exchange information from an external web service, have that information presented to certain management-level users in a readable, user-friendly fomat and added to a repository, and the information then needed to be read and approved as valid before the process would continue. One of the key design issues here was that for various reasons we could not use the SharePoint Adapter

I designed a BizTalk application which generated an xml file based on the foreign exchange currency information received from the external web service. An orchestration using a correlation set sent the information using WCF into a forms library hosted within SharePoint 2010, and then sent an email via SMTP on a dynamic port to certain management users with a link to the file. A loop ws created to continuously send the email after a certain number of minutes until the form was approved in SharePoint. They could then click on the link to open the document with an InfoPath form within SharePoint and approve the document. Upon approval, the approved xml was sent to a BizTalk WCF service hosted within IIS which (after confirming the approval based on certain business rules) sent a response to resume the same waiting BizTalk orchestration.

Whilst researching into this, I also came across R Seroter's wonderful blog post here which confirmed my design and helped speed up the development

I will try not to focus too much on the design of the BizTalk application itself as that is not the aim of this post. I will instead delve into the SharePoint and InfoPath aspects of the development more, and touch the BizTalk aspects architecturally

The steps are basically as follows:

1. Design an InfoPath 2010 form to approve the xml information
2. Create a SharePoint 2010 forms library based on the form design to accept the initial forms and store approved forms
3. Design a BizTalk orchestration which adds InfoPath processing instructions to an xml message in a message assignment shape (This is to make sure that once the xml messages are published, clicking on the xml documents will automatically open them in our InfoPath form).
4. The BizTalk orchestration sends the modified message to the SharePoint Forms library (through a WCf application which was created earlier) using a WCf-Custom adapter, as well as an email message to the management users (through the SMTP adapter) telling them the document is ready to be approved, and waits for a response (in my case, there is also a listen shape with a delay which sends a reminder email message to the management after a certain amount of time has elapsed to let them know the approval is holding things up)
5. The management users open the document using the InfoPath form and approve the document, thereby changing the status of the document to approved, and sending a confirmation to the web service
6. The web service sends a confirmation (approval) message directly to BizTalk, resuming the orchestration

PART 1 - Designing the InfoPath form.
(For this paticular implementation, make sure you have access to a SharePoint server, although with a little extra research, you can also implement similar but not exactly the same solutions without SP server (I used SharePoint 2010)

There are various ways to start off this part: for example you could start the form design from the schema based on the BizTalk message, or work backwards by basing the form design on the WCF service the form will be submitting to. I am not a SarePoint guru, but I found the second approach more foolproof, since you are sure the end result will submit successfully

1. Open InfoPath Designer (Start, All Programs, Microsoft Office, Microsoft InfoPath Designer)
2. Select 'New', and under 'Advanced Form Templates' choose the 'Web Service' option to create a form based on the web (WCF) service you wish to submit to
3. Click on 'Design Form' in the right-most panel
4. Make sure 'Submit data' is ticked on the next screen to indicate that your form will be submitting data to a service and click 'Next'
5. Enter the URL of the service WSDL (the service you wish to submit data to). If the WSDL address is correct, the wizard will expose the operations available in the service
6. Select the operation you wish to submit data to and click 'Next'
7. Enter a name for the data connection (e.g Submit To Approvals Service) and click 'Finish' (the form design view opens).
8. It is now important to remember that there are two services involved here. One is the WCF service we submit the form data to, which informs BizTalk that the form has been approved, and the other is the actual WCf service that BizTalk will call, which submits the forms to SharePoint in the first place. You should now see the schema structure expected by the WCF service we wish to submit data to in the right-side pane. Later you must ensure that your BizTalk application also uses this exact schema structure to create the xml message we want to send to SharePoint
9. Design the form interface by dragging the fields from the schema in the right-side panel. For more indepth information on this, you can try Kent Weare's post, which despite being based on an older version of SharePoint and InfoPath, is still very helpful. Also note that you can use the nifty repeating table to create the unbounded nodes from your schema quite easily. 
10. In the Controls section of the form menu, select and add any controls you wish to use on your form 
11. After all the form controls are in place, add a button to the form.
12. Right-click on the button and select 'Button Properties'
13. Select Submit as the button Action and type in a Label (e.g Approve). You can also do this using rules instead of the submit option
14. 
15. Click on 'Submit Options'
16. Click on the 'Advanced' button and fill in the page similarly to the diagram below (replace the data connection name with the name of your data connection that you created when making the form earlier, and fill in a label value to suit your preference).
17. Save the form as an xsn (form template). Note where the form template is saved to (you can use the save as dialog)
18. Click on 'File' on the menu bar and click on 'Publish'. Click on the 'SharePoint Server' button to indicate that you wish to publish the form to a SharePoint Server resource
19. Enter the address of the SharePoint site to which you wish to publish your forms. To understand this process better, you can read here or here or for some other ideas and information, you can read here
20. Select 'Form Library' and click Next. Some users of the form library might not have InfoPath installed. To enable such users to view and submit the forms in a browser, tick 'Enable this form to be submitted by a web browser'. If you don't have this option, go to Notes at the bottom of this post
21. Select the 'Create a new form library' option and click Next
22. Type in a descriptive name like 'ApproveRates' (make a note of this name for future reference), add a description (optional) and click Next
23. Add the xml/schema columns you wish to promote (make visible) in your forms library and click Next
24. Review the information, make a note of the Location and the Form Library name and click 'Publish'
25. Tick 'Open this form library' and click on the Close button
26. You should now see a preview of the new form library
27. My scenario was a SharePoint server which was part of a federated domain with the user profile service enabled. I now needed to set permissions for the BizTalk host instance to be able to add forms to the library, to avoid an error when trying to send the xml data through the SharePoint web services via BizTalk. I basically had to grant site permissions to the BizTalk user running the application pool under which the forms submission WCF service (the second service) would run. I granted permissions through the User Profile service (SharePoint Administration, 'Application Management', 'Manage Service Applications', 'User Profile Service Application'). I will not delve into this as I am not a SharePoint guru as I mentioned earlier. In a similar situation, I would assume you have a SharePoint administrator on hand to set the correct permissions
28. If your setup is simpler, you can also simply set permissions for the BizTalk user on the SharePoint site. To do this, browse to the actual site that will host your form library, click on Site Actions, Site Permissions, Grant permissions and add the BizTalk user with (at least) read and contribute permissions
29. To get the processing instructions, open the published form library by browsing to it (should be in the format http://your sharepoint server/your sharepoint site/your form library name)
30. Click on the 'Documents' tab, and click on 'Add Document' button as shown below

31. This will open up your form. Click on File and Save As to save the form. Ignore any validaiton warnings and click 'Yes' to save. Save the form as an InfoPath form (basically an xml file with InfoPath processing instructions, so that it automatically opens with InfoPath if InfoPath is installed on the user's computer) to a location you can remember
32. You have now finished the InfoPath forms design process

Notes

• If you want users of the form library to be able to submit the forms using Internet Explorer and you didn't have the tick box enabled at step 20 above, open a browser and browse to the sharepoint site in which you wish to host the form library (e.g. http:siteserver/site). You must be logged into SharePoint with an account that has enough priviledges to modify the site settings
• Click on Site Actions
• Click on Site Settings
• Under the Site Collection Administration heading, click on Site Collection Features
• Activate the SharePoint Server enterprise Site Collection Features
• Close the site
• Go back to step 19 above

Go on to Part 2 - The form submittal WCF service