How many Fault Domains?

Some weeks ago I was involved in a discussion with a customer on their design around fault domains. There was an interesting scenario which was proposed and I was asked for my opinion or validation of the aproach. Before I talk about the solution let me re-cap what Fault Domains in VSAN are.

The default storage policy in VSAN is to maintain 2 equal copies of a data object and 1 witness of metadata object, we call this Failures to Tolerate = 1. Traditionally, these objects would be automatically distributed across a separate host in the cluster allowing for the loss of 1 host, disk or other component in the cluster. This loss would only result in the VM’s compliance being negatively reported until the rebuild operation has completed and involves no VM downtime at all. In VSAN 6.0 we enabled Fault Domains which allows a customer to place hosts in a designated Fault Domain. If you are configuring Fault Domains, VSAN requires 3 and a minimum of 1 host per Fault Domain. In the primary example a Fault Domain would be a rack, hence why it is sometimes referred to as ‘Rack Awareness’. In this scenario, VSAN when distributes the copies and witness in the above example it will honour the Fault Domain configuration, meaning that no more than a single component which makes up the VM’s data will reside in a single rack. A VM with the default FTT=1 will distribute it’s data components on 1 host in every Fault Domain. This affords for the ability to lose an entire rack of infrastructure without any data availability impact to the VM. The images on the left represents data placement without Fault Domains configured. The right image represents data placement with Fault Domains configured.

Screen Shot 2015-09-06 at 4.47.56 am Screen Shot 2015-09-06 at 4.50.11 am

Configuring Fault Domains is done with a few clicks of the mouse in the vSphere Web Client.


In the scenario proposed the customer had 4 racks worth of infrastructure and had purchased only the first 12 nodes of a larger deployment to commence their project. The question was in the scenario where the cluster size was going to be 24 nodes in total however we had 12 on hand to commence the deployment. We had two choices, 3 Fault Domains with 4 hosts in each or 4 Fault Domains with 3 hosts in each. Represented by Option A and Option B below

Screen Shot 2015-09-06 at 4.56.54 am Screen Shot 2015-09-06 at 4.57.03 am

Option A                                                                          Option B

Both of these designs are perfectly valid and in some smaller customer implementations 3 racks may not even be available and hence you may not configure Fault Domains. However in this customer scenario where rack space is at a premium and the cluster was building out to a much larger scale there were was one primary benefit to the 4 Fault Domain approach.

For a moment lets look at Option A with 3 Fault Domains. Picture a full production environment which might potentially place a total 400 data components on the hosts in each rack (100 per host). Now picture the impact of the loss of a Rack A. Not only have we lost a total of 4 compute nodes but we have also temporarily lost 400 data components for the VM’s that would be affected.

Now let’s look at Option B with 4 Fault Domains. Assuming the same number of total  data components in the cluster we may be able to spread the components further across the cluster meaning that we could have only 300 data components per rack but still the same number (1200) across the entire cluster. Now think about the impact during a rack failure, A rack failure would only affect redundancy for 300 objects and VM’s rather than 4. So what, you say. Well let’s put that in context of rebuild, rebuild times and IO. Now we have 3 remaining Fault Domains available so we can immediately rebuild the failed components and maintain Fault Domain compliance, also the time taken to rebuild the data components affected in the rack failure to another host would be less as there is less data(GB) to read/write. This is often a big positive for many customers. Time-to-recovery is something many customers look at and VSAN’s distributed recovery model is a distinct advantage.


In addition, if you think of Fault Domains in the context of Maintenance Mode. If I am required to do rack maintenance or just maintenance on the hosts in a single rack I can maintain data availability during that period. In this case, the hosts in Fault Domain 4 could be used to maintain full data availability/compliance during this maintenance period.

In this case there was enough benefit for the customer to implement 4 Fault Domains rather than 3. It provided them and more desirable outcome in the event a rack failure or maintenance scenario. Now you could potentially extrapolate this scenario out to 5,6 or more Fault Domains however there will come a point where extending racks for the sake of it is impractical. Morevover, you will want to design your Fault Domains in context of the FTT policy you are likely to configure. For example, If you plan to implement FTT=2 which is 3 equal data components then you would need to distribute these across 5 Fault Domains.


In general, if you are planning for rack failures or rack maintenance in your datacenter and are planning on implementing Fault Domains, be sure to think about your host and data distribution in your own environment and plan/design accordingly.