In pricey urban locations with no available land, many organizations place their data centers on multi-use buildings' upper floors to maximize space and minimize reliability risks. Issues from above-occupied floors or other water and sewage risks can be problematic in a lower floor or basement location. Whether the roof-top data center was part of the building's initial design or created after construction, the datacenter infrastructure must be carefully managed during a roof replacement to ensure minimal operational impact.

LEDG assisted the Information and Technology Services (IT) team during the roof replacement by collaborating with various consultants and contractors assigned to the project. LEDG served as the data center subject-matter expert to oversee data center infrastructure and equipment primarily for the data center's power and cooling needs.

Services included:

• Review of construction documents to determine what roof replacement activities would impact the mechanical and electrical infrastructure.
• Identify and outline potential risks to the data center during construction.
• Provide recommendations for the information technology and construction teams to ensure the data center received the right amount of cooling and minimize data center power risks and interruptions during all phases.
• Attend weekly construction meetings to provide representation on behalf of the IT team.
• Review of construction schedules, phasing, and coordination of activities between the construction team and the IT team.
• Review the procedure documents method for all construction activities that could potentially affect the mechanical and electrical infrastructure serving the data center.

After reviewing construction plans, the roof replacement design would increase the roof's height due to new roof insulation installation. It requires installing new insulation and roofing membrane underneath each piece of equipment, including a dedicated emergency generator, four condensing units serving the In-Row cooling units, and a cooling tower serving the redundant cooling (CRAC units). LEDG worked with the general contractor to develop a phasing plan to minimize the data center's risk during each activity to ensure a successful project.

One of the top five liberal arts colleges in the U.S. needed to seamlessly move its data center from a century-old building to a more centrally located campus building with ready access to the backbone fiber and campus chilled water infrastructure.  The college used LEDG for:

• Planning and Evaluation
• Design and Engineering
• Construction Administration
• Relocation Project Management
• Commissioning

LEDG used a pre-design feasibility study to perform stakeholder interviews to create detailed user requirements that included a complete technology profile, systems integration details, cutover processes, and more. The requirements report also provided design requirements, expectations, constraints, and contingencies.

Once approved, the LEDG team evaluated the proposed data center space, including an extensive review of the architectural substitute conditions, mechanical systems, electrical and telecommunications infrastructure, fire protection, and delivered full construction documents.

The LEDG team supported the construction activities throughout the project and assisted the college information technology (IT) department in relocating critical equipment and systems. LEDG also coordinated the system startup commissioning and testing.

A lights-out data center is physically or geographically isolated from headquarters or other facilities, limiting environment fluctuations and human access. Building a separate data center also allowed the electric company to build a data center withstand an Electromagnetic Pulse (EMP) attack.  In 2019, the White House issued an Executive Order "Coordinating National Resilience to EMP" as concerns heightened over U.S. infrastructure security.

LEDG was selected to plan, design, and oversee the electric company's new data center's construction and commissioning.

Protection and Security

In a review of the security threats against the electric company's critical infrastructure, they were most concerned with High-Altitude EMP (HEMP), Intentional Electromagnetic Interference (IEMI), and ballistic (missile) threats.

Protected Space: A 6-Sided Envelope

LEDG designed the new data center using 'protected space' to define the area within the facility that would withstand an attack and developed a 6-sided envelope where the shielding created against EMP is bonded/welded together across the walls, floors, and roof. While traditional EMP protection approaches involve creating a Faraday Cage, LEDG's design used an innovative 6-sided conductive concrete to cover critical infrastructure continuously.

When considering an EMP attack, any system with electronics or a system connected to electrical power is vulnerable, making critical systems like chillers, electrical distribution, and standby generators likely to fail.  As a result, these systems need to be considered part of the protected space. For the electric utility's data center, these components were deemed critical for protection.

• Information technology (IT) systems
• Electrical systems such as the UPS, generator/ATS, and electrical distribution
• Mechanical systems
• Controls
• Monitoring and the operations center
• Security and surveillance

Building Penetration Design

The unique 6-sided conductive concrete design amplified the importance of advanced design and engineering for all entry points and penetrations into and out of the facility. In conventional construction, building penetrations for chilled water piping, electrical distribution, or telecommunications infrastructure are typically made during the installation process based on site conditions.  Because the new data center design included conductive/shielded concrete walls, roof, and slab that were welded together to create a 6-sided envelope, any penetrations made through this envelope had to be predetermined during the design process and seam-welded into the EMP concrete during construction.  LEDG utilized advanced Building Information Modeling (BIM) technologies to architect each penetration such that each point of entry was predetermined in location, size, and quantity before construction.

The HVAC design included a properly engineered Waveguide Below Cutoff (WBC) design that filters electromagnetic energy but allows air to pass.  LEDG developed a custom air-cooled chiller package that enabled the chiller and free cooling modules to function without compromising the facility's EMP protection.

Due to site constraints, the generator building was a separate protected space, and EMP protection between the data center and generator was crucial. Duct bank connections between the generator building and the data center had to be modeled before construction. All copper wiring between the buildings was installed in steel, circumferentially welded conduits.

Today, the electric grid company has a world-class data center built to withstand EMP attacks and other security risks. The highly secure, energy-efficient facility stands as the first data center in the country designed with the conductive concrete methodology tested to withstand HEMP and IEMI events.

A rapidly growing mid-Atlantic healthcare system needed to improve the redundancy and resiliency of their data center operations. The information systems and technology department provides the network infrastructure and technology platform to support the entire health network's mission-critical operations using two offsite colocation data centers. The health system determined that a third primary data center location was required to ensure continuous and reliable access.  The objective of adding the third data center is to operate each of the three facilities at 33.333% capacity. Each site can support the full technology system load in the event of a site failure.  This multi-site redundancy strategy would ensure continuous and reliable operations for the entire health network.

The hospital chose LEDG to develop recommendations and a strategic plan for the new data center's optimum location.  

The engagement included:

• An assessment of the hospital's current data center portfolio, its segregation and utilization.
• Review of current data center drawings, equipment profiles, growth calculations, reports, and policies.
• Collaboration with key stakeholders to develop requirements for the 3rd data center that represented the high-density computing profile, growth expectations, risk tolerance, geolocation, sustainability, and more.
• Site selection work through the onsite evaluation of the sites proposed for the 3rd data center, including architectural, mechanical, electrical, fire protection, telecommunications, infrastructure, and equipment.
• Development of a data center strategic plan including a recommended site, conceptual design drawings, and growth projections, anticipated construction schedule, and cost projections.

Once the assessment was complete, LEDG presented the findings to the hospital's leadership to review the project approach and assumptions, relevant data, and recommendations for the 3rd data center location with a detailed budget.  

LEDG provided the health system with a final report summarizing the process, analysis, and final recommendations for the overall data center strategy. LEDG also delivered conceptual drawings enabling the health system to present the report and recommendations to the board for final approval to move forward with data center planning, design, and build.

Leaders continue to reevaluate hybrid strategies to make sure costs are aligned, and computing demands are met.  If colocation is part of the plan, a thorough evaluation of colocation options is critical. For global financial institutions, colocation can be an integral part of a distributed, integrated IT footprint for optimal scalability and resiliency. Security, connectivity, and power are equally essential to ensure data is protected 24/7 and accessed quickly and safely.

A multinational investment and financial services organization needed to relocate a major data center to a new colocation facility. LEDG provided strategy, planning, and design for 6,000 square feet of colocation space.

LEDG's services included working collaboratively with the owner and project team to develop the data center whitespace floorplan, including critical equipment placement (storage, computing, network), aisle containment, and projected growth.  Also, LEDG's team provided strategy and design for the financial institution's implementation of a spine and leaf network topology, with uplink capacities of 100Gb.

The financial organization relocated approximately 250 cabinets worth of computing, network, and storage equipment. Diverse WLAN connections from six different carriers were needed to support high-bandwidth connectivity, including 40Gb and 100Gb connectivity within the whitespace. Due to the complexity of the network infrastructure, LEDG developed detailed rack elevations and labeling documentation to ensure that the onsite installation was accurate, standards-compliant, efficient.

The telecommunications and technology infrastructure design provided by LEDG created the backbone needed for a world-class, scalable data center to replicate across the globe in facilities that best serve the financial institution's high-performance computing demands.

A global telecommunications company had a large US-based end-user customer requesting to utilize their data center space in a major northeast city to host required gear to better support their needs. The telecom owns and operates a network infrastructure in North America that includes nationwide reach to all major U.S. cities with Multiprotocol Label Switching (MPLS) nodes across the U.S., making up one of the more extensive MPLS networks of any carrier in the region.

The telecom's customer requirements called for full data center redundancy with specific load and runtime requirements coupled with a very aggressive implementation timeline. The need for redundancy focuses on how much extra or spare power the data center can offer its customers as a back-up during a power outage. Unexpected power outages are the usual overwhelming cause for data center downtime.*

LEDG provided the telecommunications company with two 160 kVA Galaxy VM UPS for full redundancy with two wrap-around bypass cabinets. Bypass cabinets are separate cabinets that allow for complete removal of the UPS system from the load while still supplying the load with conditioned power. The solution LEDG recommended met all of the customer requirements, and the LEDG team, in collaboration with technology vendors and an electrical contractor, met the project timeline.

*PonemonInstitute 2013 Study on Data Center Outages

A prestigious Mid-Atlantic university recognized that their HPC needs were expanding and becoming more valuable to the institution. They experienced a wide diversity of density associated with high-performance computing, with rack loads varying from 5kW per rack to over 20kW per rack. They also recognized that HPC workloads are heterogeneous and will change by the application. Their goal was to accommodate the workload variances to provide greater flexibility to research and HPC clients.

As a result, with the help of LEDG, the University developed a four-tiered HPC strategy to effectively accommodate their diverse research and HPC client base, allowing clients to choose the environment best suited for their needs. With the new four-tiered plan and approach, new HPC deployments can be implemented efficiently and without compromising reliability to the data center environment.

SOLUTION

Existing Data Center

Rack Density 0-10kW

The University has segregated portions of their existing data center into higher density Point of Delivery modules or PoDs for HPC. A PoD is a set of defined compute, network, and storage resources and accommodates up to 10kW per rack of density with N+1 cooling redundancy. The PoD designs provide tighter integration and better standardization across the data center infrastructure, and the containerized approach saves time, space, and costs allowing for flexibility and growth.

Colocation Facility

Rack Density 10-20kW

For rack loads that exceed 10kW per rack, the University negotiates the option of leveraging a colocation facility. Based on the rack loads, a traditional colocation environment would not be feasible, and a dedicated suite or cage may be required. To accommodate the increase in rack density, the data center requires cooling strategies like in-row cooling with containment or a rear door heat exchanger.  A colocation provider may have the capability of delivering this capacity quickly and cost-effectively. Network connection from the University's campus currently exists to many colocation providers, and additional connections could be added if required.

Containerized Data Center Solution

Rack Density > 20+ kW

Based on the current state of high-performance computing installations on college and university campuses, rack loads exceeding 20kW per rack are not typical but are increasing. HPC workloads can scale beyond 20kW quickly, and the University is prepared to accommodate these workloads as required with a prefabricated data center solution.

Because prefabricated data centers take a modular approach to design and fabrication, they are inherently scalable and can create opportunities to add capacity as needed rapidly. An alternative to the traditional data center, a container can be placed anywhere, including non-traditional data center locations. Each container design would be created to deliver the correct reliability and capacity to match the HPC workload. Direct fiber connections to the University's network would be available to eliminate latency concerns. Developing a prefabricated data center design requires close collaboration between the data center team and the research organization, and the solution can be deployed quickly when necessary.

Offsite University-Owned Locations

Rack Density < 20kW

Like option 1, offsite locations owned by the University are possible for data center build-outs for high-performance computing. Given the on-campus demand for space and the rising costs of their urban location, the University encourages data center users to consider sites outside of campus for their HPC needs.

The offsite data centers will be committed to the same innovation in design principles, reliability, and flexibility to support different rack density profiles. Like the architecture described in the existing data center, the University can deploy high-density PODs utilizing in-row cooling or rear door heat exchangers to deliver rack densities close to 20kW per rack. The University's network is also a vital asset that deploys to offsite-owned locations. The data center team can work with research organizations to validate the network's ability to support HPC workloads with minimal latency.

The University's combination of data center assets and flexibility offered in the four-tiered approach to support high-performance computing accommodates dense workloads in many different ways. They can successfully handle different research and HPC client needs quickly, proficiently, and without compromising reliability to the data center environment.

A leading New England medical center was preparing to move their data center into an offsite colocation facility built by a local third-party company. As part of their contractual arrangement, the medical center and their colocation provider agreed that the data center would adhere to the TIA-942 Tier III specifications and included additional requirements based on their preferences and reliability needs. The specifications included detailed requirements for architectural, structural, mechanical, electrical, re-protection, telecommunications systems, commissioning, ongoing facility operations, and maintenance. The medical center needed a third-party expert to review the contract requirements and validate that the design and construction of the facility were consistent with the specifications required by the contract. The project was a significant initiative for the medical center, and careful management and expertise were needed to meet organizational goals.

LEDG acted as the medical center's representative throughout the data center construction process to validate that the design, implementation, and testing satisfactorily met the contractual agreement requirements between the medical center and the third-party colocation provider.

The data center construction faced significant challenges and delays before the medical center partnered with LEDG. Once onboard, LEDG could advise the medical center on the impact the delays would have on their data center site readiness. LEDG was the medical center's advisor and acted on their behalf for any issues that arose during the design and construction process. Through the collaboration, the third-party colocation provider was able to satisfy the Tier III requirements outlined in the contract. The medical center trusted that the site was truly ready to become their organization's data center.

LEDG Advisory Services included:

• Review of design documentation and project specifications for conformance with contract requirements.
• Conducted regular site inspections to review data center construction and proactively identify potential areas of deviation from the required specification.
• Review changes to the data center proposed by the third party and advise the medical center on its impact on compliance with their Tier III requirements.
• Review all testing and commissioning data for compliance with the Tier III requirements.
• Work with the third-party company to provide recommendations on changes required to improve non-compliant data center items to meet the contractual agreement.
• Perform the data center's final review and inspection to validate that all infrastructure, equipment, systems, and processes comply with the Tier III requirements.
• Assessment of all third-party proposed data center operational guidelines, including operations, maintenance, security, change management, and more.