Federation Architecture

While creating web applications with large, multi-tenant data sets, it becomes an immediate architectural need to plan for scaling data storage.  This plan cannot be an afterthought, but must be a paramount consideration at the outset of the project.  Performance and scalability issues on the data layer can create an array of issues that each manifests themselves as a poor user experience, and ultimately damage the brand of the application.  One such approach for handling this need is to scale horizontally using a technique known as 'Sharding'.  Sharding allows one to separate the rows of storage across multiple physical databases, which enables much scalability and should result in better performance on the data layer.  This process enables one to plan for scale, build for speed and control capacity.  Sharding also allows the operational aspect of the web application to scale, increase performance, and add additional capacity dynamically to the data layer with no downtime, which is vital to a thriving user base.

Federations are individual data partitions, which have their individual scheme centrally managed by a single distribution (or federation) scheme.  That scheme defines and controls the single keying mechanism for cross-partition distribution.  While a handful of data types are acceptable for the distribution key, I like the simplicity of a bigint as defining key (of any kind actually).

The individual partitions are members of the federation, each with their own schema.  As such, they are responsible for any inclusive subset of the values in a federated table covered by the data type of the federation distribution key.  The individual can be responsible for all of the values or a range of the values giving the architecture the ability to scale dynamically to match the current need.   While each partition has its own schema, the table keys correspond to the federation scheme. A federation member may also contain tables that are not part of the federation, known as reference tables.  Reference tables can be including in results that along with federation aware data.  It is important to note that each partition controls its own schema.  As such, it may or may not match the schema of other member partitions.

When building a federation plan, a paramount decision to make is deciding value upon which value to federate. I think the best practice may be to use a value that is meaningful to the data separation you are trying to achieve. In my world, the thing that makes the most sense is the customer or tenant identifier.  This gives us the ability to centrally reference all data for querying, yet provide each customer with what amounts to a singularly responsible and sovereign data set.

While sharding is a great solution for these types of application, it is important to understand the complexity that accompanies the sharding process. Depending on the individual implementation flavor, sharding may developers handle rollbacks, constraints, and referential integrity across tables when historically those items have been handled by the database itself.   It also makes joins, global searches and other high-level insight more difficult.  Even knowing the trade-offs being made for the ability to scale data, it is hard to argue a properly executed sharding strategy for serving multi-tenant data in a web-mobile application world.  The process checks all of the boxes required by the various user stories and operational concerns.

Sharding is a good example of a core belief of mine; It really should not matter how difficult or easy, how fancy or how simple a given technique or design is. The right answer should be the right answer.  You should not over-design because one thing seems too simple, nor should you under-design because it seems too hard.  The entirety of the platform truth should become self-evident and then pursued as the goal.