ASP.NET Web API

The ASP.NET Web API framework was introduced recently into ASP.NET.  I touched on it some previously at an overview level, but would like to dig in a little more at this time to show the power of having a true Web API in your product.  

Overview

From a high level, leveraging the Web API framework in ASP.NET allows you to easily build HTTP based services geared at reaching browsers, desktop or mobile clients.  By creating a responsive layer of RESTful APIs for a product, the platform immediately becomes a much more adaptable and scalable solution.  The simple act of ensuring all 'real' application logic is handled behind a RESTful API call, an inherent separation of interests is manifested, which enables a more loosely couple application at every layer.

ASP.NET Web API Infrastructure

At its base, the ASP.NET Web API mechanism is a simple adaptation of an HTTP service.  Web API can be used along with WCF, but realistically, they are two means to create virtually the same thing.  The main aspect of a Web API layer is the ability to create an API project within a given solution that allows user stories to be executed via RESTful calls to the services.  Microsoft has a great tutorial for creating a simple Web API project as an introduction to the platform.  For those familiar with MVC, this exercise should be very straightforward and easily worked through.  In fact, as you work through this, it will probably become clear that there is very little difference here than a standard MVC application.  The main difference is that Web API uses the HTTP method, not the URI path, to select the action. But, you can also use MVC-style routing in Web API if that is more your style.

Dynamism

Leveraging an API infrastructure as a service layer for you web application gives you the ability to manifest any client on any platform without recreating application intelligence. Some of the main features of Web API allow for the creation of a dynamic service layer that can interact with multiple versions of the same platform, 3rd party vendors, customer integrations and most importantly, your own application.  A strong API will have an entry point to accommodate the same list of user stories that the main presentation layer in your application presents to a user.  The API layer should handle modeling, security, extensibility and scalability.  This allows for the consistent use of application logic and business rules across multiple consuming platforms.  API/SDK layers should be manifested by version, with all historic versions co-existing.  A single set of business logic and data access logic should persist across versions of API.  This allows forward movement without backward degradation.  By adhering to the rule of the eternal public interface, this is an achievable goal.

The 'Real' API

While the main focus of this post is the ASP.NET Web API offering, I would like to make a point regarding what an API should be in my opinion.  An API should be a complete manifestation of every user story that exists in your application.  The API should be a service layer that is not the data access layer, nor is it the main business logic layer.  It is the manifestation of User Stories.  This type of layer has been called the User Story Layer and the services contained within have been called Complex Services.  This is a logical regrouping, or coupling of specific business logic tasks to create a user story.  A user story should be completely executed within its single methodological representation, including input validation, security and access validation, activity logging and return values. All manifested user story methods should assume no knowledge of requirements, modalities, system or data flow from any client executing the method. Data access and business logic should never be directly exposed to an end user, third party or otherwise.  Data access and business logic should be separated from each other as well as from the API/SDK/USL.  This enables the ability to quickly couple existing sets of logic into a new user story without having to accommodate situational elements inside the business logic layer.  For more information on building a strong SOA, see some of my previous posts here.  For a theoretical discussion of the purpose behind SOA, check this post.  A framework that has been built with this decoupled approach of singularly responsible object residing behind globally accessible services will much more easily allow for productivity items such as Continual Integration, Continual Deployment, and fully automated testing.

Your First API Client

The short answer here is that it’s your application.  A robust API should first and foremost serve all logic to your own applications.  You have to create your API and then consume it.  This ensures that all changes made are not only accounted for in the API, but are placed there with intent.  You have to eat your own dog food.  In a web development and agile deployment environment, the ability to decouple code and responsibility of objects is the best friend you will ever have.  Approach your web design with intent and long term vision for things like scalability and performance.  Understand that you need to support multiple versions of your service layer and extend your data objects over multiple versions.  A properly architected system will ensure you are able to respond to changes in your market and your market requirements much more quickly than would be the case otherwise.

Absence

I have had a hectic couple of weeks and have not been able to focus much time on blogging. I will be back at it soon.

C# DateTime to Javascript issues with timezone

I ran into an interesting problem when serializing a C# DateTime object as JSON and returning it via a controller to javascript.  The timezone mechanism was causing my times to shift to a localized time based on the originating timezone offset to the local timezone of the web host.  Normally, that might not be a huge issue, but since my application was displaying appointments which were being pulled from a localized data store to a user in that same region, the server timezone is irrelevant and the time for the appointment has to be the same time regardless of the timezone.  After much digging around, I found this little snippet that seems to answer the problem.

function parseJsonDate(jsonDate) {
    var offset = new Date().getTimezoneOffset() * 60000;
    var parts = /\/Date\((-?\d+)([+-]\d{2})?(\d{2})?.*/.exec(jsonDate);

    if (parts[2] == undefined) parts[2] = 0;
    if (parts[3] == undefined) parts[3] = 0;

    return new Date(+parts[1] + offset + parts[2] * 3600000 + parts[3] * 60000);
};

IIS Issue - Ignoring default document

We were having some issues trying to get an Azure app to show the proper page by default when hitting the root domain.  We flailed around for a while and eventually someone found out that there is a known issue in Windows 7 Service Pack 1 that causes the Extensionless URL feature to interfere with the way ASP.NET parses URLs that would normally be handled by the Default Document setting.  This causes redirection to the Forms Authentication login page instead of the Default Document when the url is pointing to the root of the website.  The solution is to rewrite any requests made to the root of the website to a url that explicitly references the "defaultDocument".

Do this in the global.asx:

void Application_BeginRequest(object sender, EventArgs e)
{
    if (Request.AppRelativeCurrentExecutionFilePath == "~/")
    {
        // Get the defaultDocument filename from web.config.
        System.Xml.Linq.XDocument xDoc = 
            System.Xml.Linq.XDocument.Load(HttpContext.Current.Server.MapPath("~/Web.config"));

        string defaultDocumentName = 
            xDoc.Element("configuration")
            .Element("system.webServer")
            .Element("defaultDocument")
            .Element("files")
            .Element("add").Attribute("value").Value;

        // Rewrite the url.
        HttpContext.Current.RewritePath(defaultDocumentName);
    }
}

Hope this saves someone else some time.

Automated Testing

Automated testing as a strategy

Automated testing is a great and powerful tool for ensuring consistent code coverage, performing fast regression tests, validating builds and locating potential code problems.  The introduction of automated testing into your processes should give you a noted increase in productivity and QA throughput. While it is true that the automated testing will improve your coverage and productivity, it is not the answer to every 'QA bottleneck' issue in history.  Beware of jumping to conclusions about what automated testing will bring to the table and instead focus on where it gains the most value.  It will never replace human testing in its entirety.  It simply will not replace manual testing in aspects such as product exploration and environmentally varied user story testing.

Automated tests will remove much of the burden from a human resource by testing a single action or logical group of actions repeatedly.  That is the great gift of automated testing, but it cannot entirely take the place of a human with platform knowledge exploring the product for cause and effect testing. It is important to focus your testing investments where they will have the most return.

Mike Cohn created a test automation pyramid indicating a good break down of test investments.  I have taken that and applied it to the platforms we are dealing with daily.  This pyramid illustrates a healthy distribution of test investment for complete platform coverage and explains how the investment in tests should focus at the unit level and then reduce up through the application layers.

Unit Tests

Unit tests should be your highest investment in code coverage.  These tests are created by the developers as they write the corresponding code. In a SOA world, every line of code in a service should have a corresponding unit test.  This includes all methods, extensions, data contract validations, CRUD operations and authentication routines.  These should have a test created as part of the development process.  Even when a modification is made to existing service, the modification must have corresponding tests to prove it is functioning as requested.  By its nature, a unit test will inherently give your platform total code coverage at a very base level.  It is important that all unit tests deal with every possible code path within the method being tested.  Cyclomatic complexity limitation is a standard specific metric that should be enforced with scripts, tools or code reviews, but will keep your unit tests, as well as your methods, clear, concise and focused.  For the sake of focus, I am avoiding the CC soapbox for today. This flavor of automated testing gives you the most absolute code coverage and is the base for automated testing.

Integration Tests

Integration testing is created by QA automation employees and is meant to validate functionality of entire user stories.  This is where other considerations such as performance should be measured. Integration tests should be SOA tests, meaning that the integration test plays the role of the presentation layer in executing a chain of tasks comprising user stories. These tests are constructed to validate and measure larger functional items that consist of many small service calls. A good example of an integration test is a long story consisting of customer creation, order creation, payment creation, editing of account credits and debits and comparing the end result against known values for outcome, performance and A/R.  The integration test provides regression testing at the user story level and needs to be done to all USL and SDK-API methods, as they are the outer most points in the SOA.  This type of automated testing offers the most user story coverage and should comprise the primary basis for validating the quality and functionality of your platform.

UI Tests

The UI test is the process of ensuring a UI meets specifications. UI tests are created by QA automation employees and are simply a robotic replacement for a human moving through the platform and executing the elements within a UI.  UI testing is a valid and valuable platform for product stability, but is ultimately a less indicative test of total platform quality than the areas before it.

Human Testing

Human testing should be approached as primarily an exploratory testing methodology.  In this environment, a person familiar with the platform spends the time varying individual environment variables and testing the functionality of the platform.  This is the layer at which testing no longer reflects a series of repeatable actions and begins to reflect an individual creatively trying to find flaws in a system. This is a valuable test concerning ensuring that a platform will be stable when not being used correctly or encountering anomalies in the standard environment. 
If approached with the proper expectations and focus, automated testing can be the answer to many aspects of QA that seem slow and redundant.  If we ensure the proper level of investment at each area of testing, the platform coverage can be complete and result in a much faster and more thorough acceptance process.

Federation Architecture

While creating web applications with large, multi-tenant data sets, it becomes an immediate architectural need to plan for scaling data storage.  This plan cannot be an afterthought, but must be a paramount consideration at the outset of the project.  Performance and scalability issues on the data layer can create an array of issues that each manifests themselves as a poor user experience, and ultimately damage the brand of the application.  One such approach for handling this need is to scale horizontally using a technique known as 'Sharding'.  Sharding allows one to separate the rows of storage across multiple physical databases, which enables much scalability and should result in better performance on the data layer.  This process enables one to plan for scale, build for speed and control capacity.  Sharding also allows the operational aspect of the web application to scale, increase performance, and add additional capacity dynamically to the data layer with no downtime, which is vital to a thriving user base.

Federations are individual data partitions, which have their individual scheme centrally managed by a single distribution (or federation) scheme.  That scheme defines and controls the single keying mechanism for cross-partition distribution.  While a handful of data types are acceptable for the distribution key, I like the simplicity of a bigint as defining key (of any kind actually).

The individual partitions are members of the federation, each with their own schema.  As such, they are responsible for any inclusive subset of the values in a federated table covered by the data type of the federation distribution key.  The individual can be responsible for all of the values or a range of the values giving the architecture the ability to scale dynamically to match the current need.   While each partition has its own schema, the table keys correspond to the federation scheme. A federation member may also contain tables that are not part of the federation, known as reference tables.  Reference tables can be including in results that along with federation aware data.  It is important to note that each partition controls its own schema.  As such, it may or may not match the schema of other member partitions.

When building a federation plan, a paramount decision to make is deciding value upon which value to federate. I think the best practice may be to use a value that is meaningful to the data separation you are trying to achieve. In my world, the thing that makes the most sense is the customer or tenant identifier.  This gives us the ability to centrally reference all data for querying, yet provide each customer with what amounts to a singularly responsible and sovereign data set.

While sharding is a great solution for these types of application, it is important to understand the complexity that accompanies the sharding process. Depending on the individual implementation flavor, sharding may developers handle rollbacks, constraints, and referential integrity across tables when historically those items have been handled by the database itself.   It also makes joins, global searches and other high-level insight more difficult.  Even knowing the trade-offs being made for the ability to scale data, it is hard to argue a properly executed sharding strategy for serving multi-tenant data in a web-mobile application world.  The process checks all of the boxes required by the various user stories and operational concerns.

Sharding is a good example of a core belief of mine; It really should not matter how difficult or easy, how fancy or how simple a given technique or design is. The right answer should be the right answer.  You should not over-design because one thing seems too simple, nor should you under-design because it seems too hard.  The entirety of the platform truth should become self-evident and then pursued as the goal.