In 2011 we completed several renovation projects in live data centers, meaning that the construction was going on within active data center space and while IT equipment remained operational. These active renovations always prove to be equally challenging and rewarding. An unfortunate requirement associated with many active renovations is the need for a data center shutdown at some point during the construction process. A shutdown may be required for a variety of reasons and can impact some or all systems in the data center. Simple examples are:
- In an expansion or reconfiguration, an IT equipment shutdown may occur in order to relocate racks and IT equipment into new expansion areas, or into different row or density configurations. An example is a new high density “zone” with updated power, cooling, and telecommunications infrastructure could be created to support virtualized loads. Once the “zone” is commissioned, IT equipment needs to be relocated, which may require a shutdown.
- A chiller plant shutdown may occur to increase capacity, whether in total tons, pipe sizing, etc
- An electrical shutdown may be necessary to make final connections to the main building switchgear.
Given the implications of a shutdown, most customers choose to schedule these events during the least impactful times possible, and for many organizations this happens to be the week between Christmas and New Years. For those in the data center industry, “holiday” season is many times synonymous with “shutdown” season.
All shutdowns require careful planning, coordination, and scheduling. A crucial component to the planning process is to develop a contingency plan, which outlines a course of action to complete critical shutdown tasks in the event that the shutdown events do not proceed as intended. Developing and documenting a contingency plan may seem like an obvious task, but we have found many owners who do not want to invest the time required for thorough contingency plan development. Some believe that an emergency contact list will suffice.
A great example of the value of a contingency plan came during a holiday shutdown that was just completed. Due to the varying uses of the building (in addition to a data center), scheduling the shutdown was complex and, included a significant number of stakeholders. It was also extremely ambitious in the work to be completed during the downtime. The most critical task to be completed was to install redundant feeds to a new UPS into separate ends of double-ended building switchgear. The contractors were well prepared; they had the switchgear manufacturer field-verify the breaker assemblies required prior to procurement and had all breakers tested and ready for installation.
While the schedule of events for the shutdown appeared to be low risk and well identified, it was still important to develop a contingency plan, as this was the owner’s only chance to complete the work required according to their schedule. The contingency plan identified the new breakers feeding the UPS as a risk and outlined a means of completing the installation in an alternative method if required. At a high-level, this involved repurposing some existing feeds in the switchgear (moving them to spares), adjusting trip settings, testing, and proceeding with the installation.
Even though due diligence was properly followed by the contractor, once the large switchgear covers were removed during the early morning hours of the shutdown it was clear that the breaker assembly kit was not compatible with the existing switchgear. Both new breakers onsite intended to feed the UPS were now rendered useless.
Fortunately, the contingency plan identified a course of action to be followed in the event that the new breakers could not be used. The contractors, to their credit, followed the contingency plan and were able to successfully complete the installation before the end of the shutdown.
Sometimes the simplest concepts are the ones that are overlooked during data center planning. Developing a contingency plan is unfortunately one of these concepts. Whether you are an owner or a project manager working on behalf of an owner, ensure that your contractors define, document and submit their contingency plan for your approval well in advance of your shutdown. Review this plan with all project stakeholders and verify that the plan identifies risks and associated action plans. A great contingency plan is something that you never hope you have to use, but it is an effort that should never be overlooked when planning a data center shutdown.
