Hyperscale deployment: cabinet six hundred, built like cabinet one

01 / The challengeWhy hyperscale deployment is different

A hyperscale hall is built to a capacity date, and almost every decision about how the deployment runs follows from that.

The date is committed long before the hall is ready for the installers, the link count runs into the tens of thousands, and the same work is repeated across hundreds of cabinets by different crews on different shifts, weeks apart. Four things make it a different job from an enterprise fit-out.

02 / The hallNine layers, one section through the row

A hyperscale hall is easiest to understand as a section through one row: where the fibre arrives, how it is distributed, the spine and leaf layers, the compute pods that repeat across the hall, and the storage, management, pathway and power layers that serve them. Each layer is built, and recorded, in its own way.

The compute pod is the scale unit. Once its design is right, the hall is that pod repeated, so most of a deployment method is written around building one pod the same way every time, and around the shared layers (distribution, spine, pathway and power) that have to be in and proven before the first pod is loaded.

03 / The fabricA rail-optimised pod

Inside a GPU pod there are several networks, not one, and the cabling for each is planned separately.

The compute fabric carries the east-west traffic between accelerators and is usually wired rail-optimised: each accelerator in a node connects to a different leaf switch, so the eight accelerators of a node land on eight different rails. A leaf failure then costs one rail across the pod rather than every port of a node. Storage runs on its own fabric with its own leaves, out-of-band management has its own switches and pathway, and every leaf has uplinks to every spine.

The rail design is why the port maps matter so much on site. A lead in the wrong leaf port still lights, still passes a basic continuity check and quietly breaks the rail. Every port is mapped to a node and a rail before the first lead is made up, and the map is checked again at test.

04 / Speed steps100G to 1.6T on one physical layer

800G is shipping in hyperscale halls today and 1.6T is next. The lane count is what drives the connector, so the trunking that goes in now is specified for the step after the one being bought.

Parallel singlemode optics carry a transmit and a receive fibre per lane. At 100G and 400G that is four lanes and eight fibres, which sit in a 12-fibre MPO with four positions unused. At 800G the lane count doubles to eight, and sixteen fibres need a 16-fibre connector. 1.6T doubles the lane rate rather than the lane count, so it runs on the same sixteen fibres.

05 / The link budgetEvery mated pair spends budget

At 800G the optic’s loss allowance is small, and it does not grow with the speed. Every connector in the channel spends part of it.

A link from a node in one pod to a spine in the main distribution can pass through six mated pairs: patch cord to cassette, through the trunk, through the distribution frames and out again. With standard-grade connectivity at 0.5 dB a pair, six pairs and 300 m of singlemode already exceed the 3.0 dB allowance of an eight-lane parallel optic. The same channel built from ultra-low-loss components passes with margin.

This is why connector grade and cleanliness are design decisions rather than installation details. Ultra-low-loss components buy the margin that a contaminated endface, an extra patch or a re-route would otherwise use up, and at 1.6T the allowance is no larger than it was at 800G.

06 / Pre-terminatedPre-terminated, because the density demands it

At hyperscale link counts, field termination is the slowest and least repeatable part of the job.

Trunks and cassettes arrive terminated and measured, polarity is a scheme rather than a decision taken at each link, and the hall can be re-lit at the next speed step by changing patch leads.

07 / The deployment lineGoods-in to handover, in one order

The same ten steps in the same order on every cabinet in the hall.

If a crew has to work out what comes next, cabinet six hundred will not match cabinet one. A written line, with the evidence each step produces, is what keeps a hall consistent across crews and shifts, and it is what lets a supervisor see at a glance where every cabinet stands.

08 / The cabinetThe decisions inside one cabinet

A loaded GPU cabinet fills on power and weight long before it fills on rack units.

The elevation below is an example, drawn to show how the decisions inside one cabinet fit together: where the fibre field sits, how the switching and out-of-band are placed, how heavy nodes are handled and dressed, and why free rack units are not spare capacity. On a real deployment the elevation is the operator’s, and the cabinet is built to it U for U.

09 / The standardThe operator issues the standard

At this scale the operator owns the method. The installer’s job is to work inside it exactly, in every cabinet.

Six documents usually come from the operator, and each one has a matching discipline on the floor.

10 / Waves and crewsThe hall fills in waves

A hall is released in waves of pods, and the crew ramps up and down with them.

Each wave runs the same sequence: cabinet build, node install, fabric cabling, then test and evidence. Waves overlap, so in a typical week one wave is being cabled while the next is being built and the one before is under test. The crew size follows the overlap, and the programme is worked back from the date the first pods have to be live.

11 / ScaleSizing the hall

The numbers that drive a deployment come from a small model: cabinets, nodes per cabinet, and the fabrics each node connects to.

Move the sliders to size a hall. Every figure below is built from the model shown, so you can see what drives it.

Two things stand out at any size. Fibre links outnumber nodes many times over, because every accelerator has its own port on the compute fabric; and certification, at around 900 links a test-day in this model, is a real line in the programme rather than a final afternoon.

12 / Test and evidence100% certified, link by link

At handover the evidence is the deliverable: a result for every link, a photograph set for every cabinet, and records that import on the day they land.

Fibre is tested for insertion loss and with an OTDR, with launch and receive cables fitted so the first and last connectors are measured. Copper out-of-band links are certified as permanent links to TIA-568 and ISO/IEC 11801. Nothing is sampled.

13 / How Optronix helpsOne owner for the fabric and the kit

Structured cabling, containment, power distribution and white-space fit-out are our core business. We design, supply, install, certify and migrate across the full data centre lifecycle for hyperscale, colocation and enterprise clients, across the UK, EMEA and Asia-Pacific.

One scope, from goods-in to handover

• 01

  Receipt and asset capture

  Deliveries checked against the manifest, damage recorded, and every serial, tag and position captured before a box leaves goods-in.

• 02

  Cabinet and rail build

  Cabinets set out to the floor grid, bayed, levelled and bonded, then rails and cable management built to the issued elevation.

• 03

  Compute and GPU nodes

  Heavy accelerated nodes handled with lifting aids and two people, seated on their rails, cabled clear of the airflow path and powered on in order.

• 04

  Power to the rack

  A and B feeds landed into the rack PDUs, outlets mapped to the elevation, cords cut to length, retained and dressed to the correct side.

• 05

  Pathways and containment

  Tray, basket and in-row managers installed and loaded to the design fill, with bend radius kept on every fibre route.

• 06

  Fabric cabling

  Compute, storage, in-band and out-of-band networks, installed as pre-terminated structured links with polarity managed as one scheme.

• 07

  Labelling and records

  Every port, panel and lead labelled at both ends to the operator’s scheme, and the asset register and patching schedule built as the work goes in.

• 08

  Test, certify, evidence

  100% of links certified, results recorded link by link, photographs per cabinet, and the pack handed back in the operator’s format.

1. 01

   Survey and plan

   The reference design, the pod count and the first live date are enough to start: the scope, the crew shape and an outline programme come back worked from that date.

2. 02

   Mobilise and build

   Goods-in, asset capture, cabinet build and node install, by directly employed engineers with a supervisor on every shift.

3. 03

   Cable, test, certify

   Pre-terminated fabric cabling to the port maps, every endface inspected and cleaned, and 100% of links certified.

4. 04

   Evidence and handover

   Records in the operator’s format, issued wave by wave and walked through at the hold points.

Related services