Credo VP of Optical Solutions Rajan Pai presents Credo’s optical reliability products at the 2026 AI Infra Summit.
The AI Scaling Challenge
Individually, optical components don’t have a reliability issue. Transceiver MTBF (Mean Time Between Failures )runs into the millions of hours, and in a typical three-layer Clos fabric, individual optical links carry a MTTF (Mean Time To Flap) around 300,000 hours. That’s not where the problem is.
The problem shows up once that reliability gets multiplied across scale due to cascading effect of the optical links. Optical modules already account for 60% of total networking cost in an AI cluster, at roughly six transceivers per GPU and hundreds of optical links per compute node — with 1.6T is ramping into volume production now, and with 3.2T, 6.4Tand 12.8T XPO already on the roadmap. As clusters have grown, the interval between link flaps somewhere in the fabric has collapsed accordingly: from about three hours at 30MW/20,000 GPUs in 2023, to about twelve minutes at 300MW/200,000 GPUs in 2024, to under a minute at 4.5GW/3 million GPUs in 2025 (Borrill, arXiv:2603.03736). Optical reliability didn’t get worse — cluster scale simply outran it.
The Link Flap Problem
A link flap is a much bigger deal here than in a typical enterprise network because AI training is synchronous. A single unstable link doesn’t just degrade one connection — it cascades, because every GPU in that collective operation waits on it.
On a 32,000-GPU cluster checkpointing hourly, a single unstable link can waste 32,000 GPU-hours in one event, since the whole job rolls back to the last checkpoint. Across large training runs, persistent link instability — the time it takes to identify the faulty module — is estimated to cost 20–30% of total productivity.
On a brand-new 100,000-GPU cluster, assuming a conservative five-year per-module MTBF, the math puts the statistically expected first job failure within 26 minutes of power-on. One concrete example from the field: a single flapping transceiver reportedly cost a 50,000-GPU OpenAI training job roughly 25% of its throughput.
The net effect is a stubborn utilization ceiling: clusters without proactive optical diagnostics tend to plateau at 65–75% GPU utilization; with proactive telemetry, that climbs past 90%. That ~25-point gap isn’t a mystery. It comes from network-driven stalls, and it has a specific fix: proactive diagnostics.
The root causes behind the flaps are mostly mundane: thermal sensitivity, connector contamination, ESD damage during install, vendor/firmware incompatibility, manufacturing defects, and poor connector hygiene. None of these are exotic. They’re the kind of thing a mature monitoring architecture should catch long before they cascade into a training outage.
Six Architectures, One Inconsistent Telemetry Story
At least six optical architectures are in production or ramping today — fully-retimed optics (FRO), linear-drive pluggable optics (LPO), linear receive optics (LRO), active optical cables (AOC), co-packaged optics (CPO), and near-packaged optics/optical I/O (NPO/OIO).
Each trades power against diagnostic depth differently. FRO runs hottest, at 25–30W, but carries full CMIS+CDB telemetry and full diagnostic capability. LPO cuts power roughly in half, to 10–13W, but its telemetry depth is limited. CPO gets down to single-digit watts (8–9W) but its diagnostics are still evolving, and adoption for AI remains under 1% as of 2026.
The pattern across this landscape is consistent: the industry has optimized for power and reach on all six architectures, and only inconsistently for the ability to tell, in real time, whether a given link is actually healthy.
Advanced Telemetry — The Solution
Solving the link flap problem starts with identifying what today’s standard diagnostics can’t do. Even the newer VDM (Versatile Diagnostic Monitoring) layer defined in CMIS, sitting on top of legacy DDM, (Digital Diagnostic Monitoring) mostly reports what already happened on a link — not whether it’s healthy, degrading, or minutes away from taking a job down.
Closing this gap has been the focus of ZeroFlap optical transceivers for AI interconnect, and featuring our PILOT telemetry software. The starting point is a health score: a single number that tells an operator immediately whether a link is running in excellent condition, degrading, or critical enough to pull offline before it takes a training job down with it.
_
Getting that score right took real study, since it must reflect the metrics that can predict a flap rather than just the ones that are easiest to measure. FEC bit-error-rate is one of them — PILOT tracks how quickly that error rate is climbing, not just whether it has crossed a fixed threshold, so a warning reaches an administrator while there’s still time to swap the module before the flap happens.
Because most optical instabilities are transient, ZeroFlap transceivers keep a timestamped log of every one, which is what makes it possible to tell a module that flaps constantly apart from one that flapped once and recovered — a distinction that matters enormously when deciding what actually needs replacing. PILOT also watches for multi-path interference, catching the receive-power fluctuations that dust or debris on a connector typically cause, well before they show up as an outright failure.
None of this stops at one end of a link. A ZeroFlap device’s firmware and diagnostics can be managed from a link’s far end without separate physical access to it, and, more importantly, a host and its remote module partner can communicate about link health. If receive power starts dropping on one side, the fix can be as simple as boosting transmit power on the other — PILOT is built to make that exchange automatic, rather than something an engineer needs to notice and intervene on by hand.
Put together, this is what turns a rack of optics from a collection of individually-monitored parts into a cluster with real, end-to-end visibility into which links are healthy and which ones need attention.
Field Validation
A two-month ZeroFlap PILOT cluster ran across 128 modules, switch-to-GPU, comparing a reactive baseline against modules reporting this predictive telemetry — tracking flap rate, mean time to detect, false-swap rate, and utilization. The predictive-telemetry group reached target utilization through early-warning replacement of at-risk modules, not a slow organic ramp: catching weak links before they became job-stopping events, rather than after and analyzing the system performance over the entire period
The Standardization Imperative
A proprietary telemetry scheme, however good, only solves the problem for whoever built it. With six competing optical architectures and dozens of vendors shipping into the same clusters, a health score or event log that only works on one company’s modules doesn’t help an operator much when the rack next to it is running someone else’s transceiver.
That’s the case for treating this as a standards effort rather than a product feature, and it’s the approach Credo has taken: proposing the PILOT telemetry model — health score, FEC bin, event log, MPI detection, remote firmware management — as an OCP Optics Telemetry Software Specification, so any transceiver vendor can implement the same interface rather than inventing its own.
In parallel, Credo is contributing that same model into Community SONiC (sonic-net/SONiC), the open network operating system many of these clusters already run, through a dual-track process that keeps the OCP-defined interface and its SONiC implementation moving together instead of drifting apart.
The design asks for as little disruption as possible on purpose: no changes to SAI, no new YANG models, no database schema rework.
Driving Toward a Reliable Optical Layer for AI
A reliable optical layer for AI infrastructure requires action at three levels, moving together.
At the transceiver level, that means implementing OCP-aligned telemetry as a standard, not a vendor-specific extension — health scoring, FEC stats, event log, MPI, remote FW management — on a framework that holds across 1.6T today, 3.2T, 6.4T , and 12.8T XPO after that.
At the NOS integration level, Community SONiC provides the open platform, with a replicable contribution model any vendor can follow and a consistent Redis STATE DB schema across all of them.
And at the operations level, it’s a genuine shift in posture: from reactive — replace after failure — to predictive — replace on threshold — turning a default monitoring set into decision support at scale.
That last shift is the one that eventually shows up on a utilization graph: from roughly 70% GPU utilization to 90%+ with proactive optical management. AI cluster scale has created a reliability challenge that reactive management, on its own, was never going to keep pace with. Advanced telemetry — and the standards work to make it universal rather than proprietary — is what is needed to close that gap.
