Optronix, a part of the GS Group

11Operations guide

Running a data centre: operations, resilience and uptime

Commissioning a facility is the start line, not the finish. Uptime is not something you buy with redundant equipment; it is earned every day through disciplined operations. This guide covers the operational stack, what the nines actually cost in downtime, the disciplines that separate a resilient site from a fragile one, and how incidents are really handled.

A vendor-neutral, educational guide. Availability figures and frameworks are referenced from the Uptime Institute, ASHRAE and common industry practice; see the notes at the end.

A monitoring trace: telemetry runs steady, crosses the alarm threshold, is answered by a rehearsed response and returns to normal; the gap between alarm and recovery is marked as MTTR. Telemetry / supply air Live Alarm threshold Alarm 03:12 Restored MTTR UPS A, normal CRAH 02, normal Hall 2 inlet, alarm cleared
Fig. 11 / Monitoring trace
Dwg
OX-RES-11
Rev
4.0
Scale
NTS
  • ~66%Of significant outages involve human error: almost always a skipped or absent procedure, not bad luck
  • 1.6hrsAnnual downtime at 99.982%, the availability most enterprises target
  • 5minAll the downtime five nines (99.999%) allows in a whole year
  • MTTRThe one availability lever operations owns outright: how fast you recover, not how much plant sits underneath
  • 24/7Monitoring, response and on-call never stop, including 3am on a bank holiday

01Uptime is operated, not bought

The building gives you power, cooling and space, but that is the floor, not the finish.

Whether your estate actually stays available depends on how it is operated, day in, day out. The industry's own data is clear: most major outages are not plant failures but operational ones. A change made without controls, an alarm no one answered, a dependency nobody documented. That operational layer, not the M&E underneath it, is where availability is won or lost, and it is what this guide covers.

Facility

The facility is the floor

Power, cooling and space are the facility's M&E. They set the conditions, but they do not keep your estate running. Everything above the floor (hardware, connectivity, changes, monitoring) is operations, and it is where the day-to-day risk lives.

People

People are the variable

The single largest cause of avoidable downtime is human error during change and routine work. Procedures (MOP, SOP, EOP), training and supervision are the controls that turn a good environment into a reliable one.

Gaps

The risk is in the gaps

Outages cluster around the seams: an undocumented change, a cross-connect without a diverse path, a monitoring blind spot, a single engineer who knows how the rack is wired. Operations is the discipline of closing those gaps before they find you.

Recovery

Recovery is a capability

Failures are inevitable; extended outages are not. Whether a fault is a 30-second blip or a six-hour incident is decided by monitoring, escalation and rehearsed response, long before the alarm sounds.

02The operational stack

The facility hands you power, cooling and space. Operations is everything done on top of that to keep your estate available: the hardware, the connectivity, the capacity, the monitoring and the people.

The estate is only as available as the weakest of these layers, so each is a domain an operations team runs around the clock. Hover or tap a layer.

  1. A

    Hardware and rack operations

    Installs, moves, swaps, break-fix and RMA handling: bringing equipment in, keeping live hardware healthy and replacing what fails, correctly and to schedule.

  2. B

    Connectivity and cabling

    Cross-connects, patching, port management, labelling and documentation. Power keeps equipment alive; the network makes it useful, and undisciplined cabling is a slow-motion outage.

  3. C

    Capacity and asset management

    Tracking space, power and port headroom against an accurate, live asset register, so you never strand capacity or, worse, run out of it mid-deployment.

  4. D

    Monitoring and alerting

    Watching the environmental and power telemetry the facility exposes (temperature, humidity, load, access) and acting on it: spotting the deviation and escalating while there is still time.

  5. E

    Physical security and access

    Access control, escorting, CCTV review and audit logging. Who is on the floor and what they touch is part of availability and compliance, and a discipline in its own right.

  6. !

    Single points of failure

    The job is to find the link that is not resilient: the cross-connect without a diverse path, the undocumented dependency, the single competent engineer. Unmapped single points of failure are where “resilient” estates still fail.

03Where outages are won and lost

Operational maturity is the gap between a site that has good equipment and one that is run well.

Not every discipline carries the same weight. The bars below show how often each one decides how an incident ends, rather than whether it happens at all. Anything you cannot evidence with a document or a record is a gap, not a strength.

  1. Monitoring and alertingCritical
  2. Procedures (MOP/SOP/EOP)Critical
  3. Change controlCritical
  4. Connectivity and cablingHigh
  5. Incident response and on-callHigh
  6. Capacity managementHigh
  7. DCIM and asset accuracyHigh
  8. Failover and DR testingMedium
  9. Training and competencyRising

Score your own estate against these

There is a weighted self-assessment in our managed operations guide: nine disciplines, four levels each, and it shows which of your gaps cost the most.

Score your estate

Monitoring and procedures sit at the top on purpose: you cannot respond to what you cannot see, and you cannot act safely without a method. The most common failure pattern is not missing redundancy. It is redundancy that was never tested, or a procedure that existed only in one engineer's head.

04What the nines really cost

Availability is quoted in nines, and the gap between them is large: each extra nine cuts your allowable downtime roughly tenfold.

Knowing what a target actually permits, in minutes per year, is the only honest way to decide what resilience you need and what it is worth paying for.

Interactive / Downtime budget

What an availability target allows

Pick a target to see the downtime it permits per year, month, week and day.

99.982% availability allows

1.58 hrs

of downtime a year: common enterprise availability target

Per year1.58hrs
Per month7.88min
Per week1.81min
Per day16s

Bars use a logarithmic scale, so each step down is roughly ten times less downtime. The figures are the arithmetic of availability, not a guarantee.

AvailabilityDowntime per yearDowntime per monthTypical context
99% (two nines)3.65 days7.3 hrsNon-critical, development and test
99.9% (three nines)8.76 hrs43.8 minStandard business systems
99.982%1.58 hrs7.9 minCommon enterprise availability target
99.99% (four nines)52.6 min4.4 minCritical production
99.999% (five nines)5.26 min26 secMission-critical, fault tolerant

A target only holds if the operations behind it do. Five nines means roughly five minutes of unplanned downtime in an entire year, which takes a resilient facility and mature operations, and operations is the part most often under-invested in. Note the distinction between planned and unplanned downtime: a well-run estate schedules disruptive work into change-controlled maintenance windows, so the goal becomes zero unplanned outage.

How often

MTBF

Mean time between failures: the expected interval between faults on a component. Higher is better, and it is improved by quality equipment and preventive maintenance.

How fast

MTTR

Mean time to repair or recover: how long it takes to restore service once something fails. This is the number operations controls most directly, through spares, monitoring and rehearsed response.

Which kind

Planned is not unplanned

Disruptive work scheduled into a change-controlled maintenance window is planned and managed. Counting it as downtime confuses it with the real enemy: the goal is zero unplanned outage.

05The operational disciplines

Reliable sites are not reliable by accident. They run a set of repeatable disciplines that catch problems early, make every action safe and repeatable, and keep capacity ahead of demand.

These are the practices that turn a redundant design into a resilient operation.

01

Change control

No change to a live system without review, approval, a method statement and a rollback plan. The single highest-leverage control against self-inflicted outages, because most outages are self-inflicted.

02

MOP, SOP and EOP

Method, standard and emergency operating procedures: the written, rehearsed scripts for routine work and for the bad day. When the alarm goes at 3am, you follow the EOP; you do not improvise.

03

Capacity management

Tracking space, power and port headroom so you never strand capacity or run out of it mid-deployment. Stranded capacity is wasted spend; exhausted capacity is a stalled project.

04

Asset lifecycle management

Knowing what every asset is, where it sits and where it is in its life, from receipt and deployment through refresh to secure decommissioning. The backbone of capacity, support and audit.

05

Connectivity and cabling discipline

Structured cabling, change-managed patching, labelling and accurate records. Tidy, documented connectivity is fast to work on and safe to change; the alternative is an outage nobody can trace.

06

DCIM and documentation

An accurate, live record of every asset, circuit and connection. Decisions made on stale data cause outages; DCIM and an up-to-date asset register are the operational source of truth.

When something goes wrong: the incident lifecycle

Detection to recovery is a rehearsed sequence, not a scramble. Every minute saved here comes off your MTTR, and it is the difference between a near miss and a reportable outage.

  1. 01

    Detect

    Monitoring catches the deviation (a temperature trend, a failed unit, a power anomaly), ideally before it affects the load.

  2. 02

    Alert and triage

    The alarm reaches the right person on a clear escalation path, and severity is assessed against defined thresholds.

  3. 03

    Respond to procedure

    The relevant EOP or SOP is executed (isolate, fail over, switch to the alternate path) methodically, not improvised.

  4. 04

    Stabilise

    Service is protected or restored, the fault is contained and the site returns to a known-good, redundant state.

  5. 05

    Restore and verify

    The failed component is repaired or replaced, redundancy is re-established and the fix is verified under load.

  6. 06

    Review

    A blameless post-incident review feeds back into procedures, training and design, so the same fault cannot cause a second outage.

06How Optronix helps

Operations is the longest stage of the data centre lifecycle, the years between building a facility and decommissioning it, and the one where reputations are made or lost.

We support clients across the operational disciplines, with smart hands and live-operations cover across the UK, Europe and worldwide, whether you run the site yourself or want us to carry the load.

  1. 01

    Operational readiness and audit

    We assess your estate against the disciplines above (procedures, change control, monitoring, capacity, cabling and single points of failure) and give you a prioritised plan to close the gaps.

  2. 02

    Smart hands and remote operations

    Eyes and hands on the floor for the lights-out site: installs, swaps, audits and break-fix, to your procedures and to schedule.

  3. 03

    24/7 monitoring and response

    Round-the-clock cover with clear escalation and rehearsed incident response, so an alarm at 3am is met with a procedure, not a scramble.

  4. 04

    Capacity, asset and cabling support

    Capacity tracking, asset register and DCIM accuracy, and structured cabling work, keeping your estate ahead of demand and audit-ready.

Operations support

Tell us where it hurts.

Coverage gaps, an upcoming maintenance window, a site that needs eyes on it: we will put the right operational support around it, on the ground wherever it is.

Share this guide with your team. Share on LinkedIn

Sources and notes

  • Uptime Institute Annual Outage Analysis: human error as a factor in the majority of significant outages, and the planned and unplanned distinction.
  • Availability and downtime figures are the standard arithmetic of the nines (for example 99.999% is about 5.26 minutes a year).
  • Common industry practice for change control, MOP/SOP/EOP, DCIM, and capacity and asset management.