Thursday, August 27, 2015

QuickBooks Web Connector with WCF services

I have been working a project to try and get some backend data plugged into QuickBooks so we can push that data into QuickBooks. We are using desktop QuickBooks hosted by another provided so our integration options are limited. We ended up using the QuickBooks Web Connector. First let me say that documentation on this front is limited and what you can find is normally old. Here is some of the reference we were able to find:
Web connector overview
QuickBooks error codes
Helpful UI for how to construct QuickBooks API calls
Web connector SDK
QBWC Dev Guide
You will need to create a QWC file and here is a QWC Example.
One of the biggest issues we ran into was all the documents and examples we could find were based on the older ASMX services. Because of this we struggle for a bit on understanding how the service needed to be attributed. So here is what our WCF service contract ended need to be set as.
 

   [GeneratedCode("wsdl", "4.6.57.0")]

   [WebServiceBinding(Name = "QuickBookConnectorService", Namespace = "http://developer.intuit.com/")]

   public class QuickBookConnectorService : QBWebConnectorSvc, IQuickBookConnector

   {

       /// <summary>Provides a way for web-service to notify QBWC of it's version. This version string shows up in 

       /// the More Information pop-up dialog in QBWC.</summary>

       /// <returns>The <see cref="string"/>.</returns>

       [OperationContract(Action = "http://developer.intuit.com/serverVersion")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override string serverVersion()

       {

       }

 

       /// <summary>Optional callback allows the web service to evaluate the current web connector version and react to it.</summary>

       /// <param name="strVersion">The version of the QB web connector supplied in the web connector's call to clientVersion</param>

       /// <returns>The <see cref="string"/>See QB Dev guide for details on return values</returns>

       [OperationContract(Action = "http://developer.intuit.com/clientVersion")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override string clientVersion(string strVersion)

       {

       }

 

       /// <summary>The authenticate.</summary>

       /// <param name="strUserName">The username the connector should be running as to authenticate.</param>

       /// <param name="strPassword">The password the connector should be running as to authenticate.</param>

       /// <returns>The <see cref="string[]"/>Must contain 2 elements. 1) NONE, NVU, BUSY, string.empty or Company file name. 

       /// 2) Value noted how long to postpone the update 

       /// 3) Sets lower limit on how often QBWC can call 

       /// 4) Sets the min run seconds</returns>

       [OperationContract(Action = "http://developer.intuit.com/authenticate")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override string[] authenticate(string strUserName, string strPassword)

       {

       }

 

       /// <summary>Tells your web service about an error the web connector encountered in its attempt to connect to QuickBooks or QuickBooks POS.</summary>

       /// <param name="ticket">The ticket from the web connector. This is the session token your web service returned to the web connector's

       ///  authenticate call, as the first element of the returned string array</param>

       /// <param name="hresult">The HRESULT (in HEX) from the exception thrown by the request processor</param>

       /// <param name="message">The error message that accompanies the HRESULT from the request processor.</param>

       /// <returns>The <see cref="string"/>.</returns>

       [OperationContract(Action = "http://developer.intuit.com/connectionError")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override string connectionError(string ticket, string hresult, string message)

       {

       }

 

       /// <summary>The web connector's invitation to the web service to send a request.</summary>

       /// <param name="ticket">The ticket from the web connector. This is the session token your web service returned to the web connector's

       ///  authenticate call, as the first element of the returned string array.</param>

       /// <param name="strHCPResponse">Only for the first sendRequestXML call in a data exchange session will this parameter contains response 

       /// data from a HostQuery, a CompanyQuery, and a PreferencesQuery request. This data is provided at the outset of a data exchange because

       /// it is normally useful for a web service to have this data.In the ensuing data exchange session, subsequent sendRequestXML calls from 

       /// the web processor do not contain this data, (only an empty string is supplied) as it is assumed your web service already has 

       /// it for the session. </param>

       /// <param name="strCompanyFileName">The company file being used in the current data exchange.</param>

       /// <param name="qbXMLCountry">The country version of QuickBooks or QuickBooks POS product being used to access the company. .</param>

       /// <param name="qbXMLMajorVers">The qb xml major vers.</param>

       /// <param name="qbXMLMinorVers">The qb xml minor vers.</param>

       /// <returns>The <see cref="string"/>Either string of "NoOp" to tell the QBWC to pause or a valid qbXML string</returns>

       [OperationContract(Action = "http://developer.intuit.com/sendRequestXML")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override string sendRequestXML(

           string ticket,

           string strHCPResponse,

           string strCompanyFileName,

           string qbXMLCountry,

           int qbXMLMajorVers,

           int qbXMLMinorVers)

       {

       }

 

       /// <summary>Returns the data request response from QuickBooks or QuickBooks POS.</summary>

       /// <param name="ticket">The ticket from the web connector. This is the session token your web service returned to the web connector's

       ///  authenticate call, as the first element of the returned string array.</param>

       /// <param name="response">Contains the qbXML response from QuickBooks or qbposXML response from QuickBooks POS.</param>

       /// <param name="hresult">The hresult and message could be returned as a result of certain errors that could occur when 

       /// QuickBooks or QuickBooks POS sends requests is to the QuickBooks/QuickBooks POS request processor

       /// via the ProcessRequest call.If this call to the request processor resulted in an error (exception) instead of a 

       /// response, then the web connector will return the corresponding HRESULT and its text message in the hresult and message 

       /// parameters.If no such error occurred, hresult and message will be empty strings.</param>

       /// <param name="message">See above under hresult.</param>

       /// <returns>The <see cref="int"/>A positive integer less than 100 represents the percentage of work completed.</returns>

       [OperationContract(Action = "http://developer.intuit.com/receiveResponseXML")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override int receiveResponseXML(string ticket, string response, string hresult, string message)

       {

       }

 

       /// <summary>Allows your web service to return the last web service error, normally for display to the user, before 

       /// causing the update action to stop.</summary>

       /// <param name="ticket">The ticket from the web connector. This is the session token your web service returned to the web connector's 

       /// authenticate call, as the first element of the returned string array.</param>

       /// <returns>The message <see cref="string"/> describing the problem and any other information that you want your user to see</returns>

       [OperationContract(Action = "http://developer.intuit.com/getLastError")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override string getLastError(string ticket)

       {

       }

 

       /// <summary>Tells your web service that the web connector is finished with the update session</summary>

       /// <param name="ticket">The ticket from the web connector. This is the session token your web service returned to the web connector's 

       /// authenticate call, as the first element of the returned string array</param>

       /// <returns>The <see cref="string"/>Specify a string that you want the web connector to display to the user showing the status of

       /// the web service action on behalf of your user.This string will be displayed in the web connector UI in the status column.</returns>

       [OperationContract(Action = "http://developer.intuit.com/closeConnection")]

       [XmlSerializerFormat(Style = OperationFormatStyle.Document, Use = OperationFormatUse.Literal)]

       public override string closeConnection(string ticket)

       {

       }
Consolibyte has some good information on his wiki as well.






Tuesday, August 25, 2015

Installing Tridion 2013 SP 1 on Hyper-V

I recently needed to install Tridion on Hyper-V. To do so I found a great series of blog posts on the Tridion website. The series starts with setting up a VM image instead of a Hyper-V. Really there is no difference (you install the OS on the image). One issue I did run into with Hyper-V though was my local network started to run a bit slower. I did some digging and found out it was because when the network bridge was setup for the Hyper-V image the priority order of the adapters put the Hyper-V image first. IT Writing has a great blog post on how to fix this.

Once you have Hyper-V ready you can follow the post step for step to get everything installed. I needed once change in that I already had a Content Management database setup. I wanted to move my database from production down into this Hyper-V instance so I did not need to setup the database from scratch.Here is what I did and a few of the tweaks I needed to make.

After installing I would try and open up the Tridion CM website. However, my login always failed. Looking into the event viewer’s security log I found the following:

An account failed to log on.

Subject:
    Security ID:        NULL SID
    Account Name:        -
    Account Domain:        -
    Logon ID:        0x0

Logon Type:            3

Account For Which Logon Failed:
    Security ID:        NULL SID
    Account Name:        MTSUser
    Account Domain:        WIN-M3213T5Q594

Failure Information:
    Failure Reason:        An Error occured during Logon.
    Status:            0xC000006D
    Sub Status:        0x0

Since the Hyper-V image was running outside of a domain controller I did some digging and found a Microsoft KB article that fixed the issues. My resolution was method 1.

The other thing I had to do was update the Trustees table in the Content Manager database. Keep in mind that I am restoring an existing database. That existing one is on a domain controller so the MTSUser account that was created was linked to a different domain. So in the Trustees table find your MTSUser and in the name column update the domain part of the domain to whatever your Hyper-V’s machine name is (<machine name>\MTSUser). This assumes that during your install you said your system account was MTSUser.

After these steps I was able to login and all was good.

Friday, May 15, 2015

Entity Framework Code First – LoadFrom Error

 

I have been playing with Visual Studio 2015 RC and Entity Framework Code First. In the processes of doing so I ran into this error.

Exception calling "LoadFrom" with "1" argument(s): Invalid URL…….

This started happening when I upgrade to version 6.1.3. At first I thought this was a file path issue as I also moved where my project was located. That was not it through. I found a couple similar issues doing some searching but the error message was a little different.

Visual Studio 2015 RC Entity Framework 6.1.3 Migrations Error

Code First Migrations StartUp Project Does Not Reference The Project Contains Migrations

The first error I got was about file length, but I had seen that in Visual Studio before so I just moved where my solution was located. Once I moved the location I got this new error about invalid URL.

The stackoverflow thread though did point me to a GitHub thread with good info on it. The thing that solved it for me was two things.

1- Downgrade to version 6.1.1

2 – Update the EntityFramework psm1 file with the contents of this file. Note: You can just replace the file. You have to open the file up and replace the contents. If not when you restart Visual Studio you will get a signing error.

3 – Restart Visual Studio.

Friday, April 3, 2015

Windows Workflow, child workflows and parallel loops

I recently needed to create a pretty complex workflow to process a large volume of records as quickly as possible. We needed to process 500k records in an hour. We have a scheduled process that triggers the initial workflow that looks for scheduled work. When work is found a call is made out to another system to load the data we need per order (so this can be 1-X order records). Each order record can have 1 to X  number of customer records we need to process. Along the way we have a few user approvals we wait for. In trying to create a Workflow process for this we ran into a few challenges and learnings. Hopefully, this helps some others out there.

At a high level here is what we put together. 

image

The idea here is the Management Workflow is the workflow that is fired off by the schedule. It then can fire off the processing workflow. The green workflows can have multiple instances of them created by its parent.

The Processing Workflow is responsible for finding all the customers that need to be processed for a given order.

The Send workflow is responsible for packaging up all the data for a given customer (calls to CMS system or any other external system) and sending that data to the delivery system.

With these processes able to spin up multiple instances and load balances via IIS we are able to processes thousands of records a second. 

Since these are Windows Workflow Services hosted in IIS we load balance new workflow instances across servers. This approach allows us to scale both out and up to deal with demand. There are some things to keep in mind when setting Windows Workflow up this way though.

Duplex Communication

The first thing we needed in this model is a way for the Processing workflow to spin up child (Send) workflows and to have those child workflows report back on there status.

We used an approach called out on MSDN for Parent-Child Workflow Pattern Using Durable Duplex. Based on this code we were able to create our setup and get this parent child relationship working. To make this work you need to make sure you get your WCF bindings configured correctly and make sure your workflow objects are setup with Callback Correlation correctly. The easily way to get start on this is to download the example (direct link) and really review it.

Throttles

One of the main issues we ran into once we got this setup and running was throttling issues. In each workflow we had a couple Parallel ForEach loops and some async activities. This was one of those situation where we actually slowed ourselves down by trying to go to fast. The workflow would spin so many parallels up and connections to other servers that we killed our active connections causing an app pool reset.

To solve this we had to do a couple of things. First, realize that just because you can parallel does not mean you should parallel. We reduced the number of parallel for each loop we had or manually throttled how many could be created at one time (wish Workflow designer gave you a setting for this). Second, we realized that in production there is some tuning that needs to be done on WCF (these are all Windows Workflow services so WCF). There is a great blog post for Tweaking your WCF apps for high throughput workloads that helped. It calls out how to troubleshoot this issue and the config changes you cam make for MaxConcurrentCalls, MaxConcurrentInstances and MaxConcurrentSessions.

MaxConcurrentInstances (when you follow the steps in the blog) will stop additional ones from spinning up. But we also needed to limit how many items we passed into the parallel so it did not spin up to much at a time. We did this by using a while and simply picking a certain number from our collection and passing those to a parallel for each. As long as the while still had items to pick it kept going back to the parallel.

Tracing

Now that you have these workflows up and running how do you keep track of what they are doing, what state they are in and if the child workflows are healthy? If the child workflows are calling back to the parent what do you do when one dies on you?

I am sure there are a number of ways to tackle this but here is what we did. We used Windows Workflow Foundation tracking we created a custom tracking service so we can track and log the data we need about the workflow to our database. This allows us to track each state of the workflow we care about. So now we can log when a workflow goes into an aborted or unhandledexception state. Keep in mind that if you track everything this will track A LOT of data and write A LOT to the database. So in production it is important to create a custom tracking profile to limit what is logged to only what is required.

Now that you have your workflows logging information to trace you can have your workflows start to monitor that data. In our case we have the parent workflow keep track of what workflows it spins up. Then for each workflow it spins up in goes into a pick activity. One pick branch is triggered when the child calls back. The other pick branch has a heartbeat (a delay followed by a check on the child’s workflow state). If the child workflow state is found to be in an bad state (based on the state logged by the tracker) a flag is set to mark that workflow as unhealthy and the pick branch’s action is triggered. If the child workflow is healthy the heartbeat loops and wait for a configured amount of time before firing off again. If the child workflow stays healthy and calls back the first pick branch’s action is triggered and we go on about our marry way.

Now you have an approach to creating a parent child workflow and monitoring the states of the children. Next up is moving this solution to a load balanced environment (post in progress).