The short answer

Here is the whole comparison in a few sentences. OSPF and IS-IS are two systems that do one job: they keep every router in a network holding the same up-to-date map, so each router can work out the best road for your data. OSPF is the one most companies run, and the one almost every engineer learns first. IS-IS is its older cousin, and it is still the favourite inside the very large backbone networks that carry the internet itself. If you are choosing what to learn: OSPF first, IS-IS second if you aim at carriers, cloud giants or high-speed trading firms. If you are choosing what to run: OSPF for a normal company network, IS-IS for a huge backbone, and — in a modern data centre — possibly neither. Everything below is the why, in plain words.

Picture two delivery firms using the same road atlas. Both send drivers out to report every new road and every closure. Both pin those reports on the same wall map. And both plan routes with the same rule: take the cheapest total road. The difference is in the paperwork. One firm files everything in folders that belong to one numbering system. The other writes each fact on its own labelled slip, so a brand-new kind of fact is just a new slip in the same folder. That filing difference — not the map, and not the route maths — is most of what separates OSPF from IS-IS.

In plain English: Router: a junction box that moves data between networks and decides which road each packet takes. Routing protocol: the system routers use to build and update their shared map. IGP (interior gateway protocol): a routing protocol used inside one organisation's network — OSPF and IS-IS are both IGPs; BGP is the one used between organisations. Link: one connection between two routers — one road on the map. Link-state: every router shares what its own links look like, so all routers end up holding the same map. Shortest-path maths (SPF, also called Dijkstra): the calculation each router runs over the map to find the cheapest route — the same kind of maths a satnav uses. Area / Level: a way to cut a big map into smaller pieces so each router does less work. TLV (type-length-value): a labelled slip inside a protocol message — a name, a size, then the fact itself — which makes a protocol easy to extend.

First, what is the same — and it is most of it

Start here, because most arguments about these two skip it. Both begin the same way: neighbouring routers introduce themselves with small "hello" messages and agree to work together — engineers call that agreement an adjacency (in plain words: a neighbour partnership). When anything changes — a link fails, a new one appears — the routers that saw it write a short update note and pass it to their neighbours, who pass it on, until every router has it. That passing-along is called flooding, and it is really just everyone making sure nobody is working from an old map. OSPF calls these notes LSAs; IS-IS calls them LSPs. Different names, same thing: map updates.

Every router keeps all the notes it has received in one pile — its copy of the map (engineers call it the link-state database). Because everyone floods every note to everyone, every pile ends up identical. Then each router runs the same shortest-path maths over its own copy, giving each road a price tag (the cost — fast, reliable roads get low prices) and picking the cheapest route to everywhere. When the map changes, everyone re-runs the maths and settles on the new routes; that settling is called convergence. An engineer who truly understands one of these protocols already understands about eighty percent of the other. Mostly the vocabulary changes, not the ideas.

That also settles the most common argument: neither protocol is magically "faster" in a normal design. How quickly a network settles after a failure depends on how fast the failure is noticed, how fast the notes flood, how the re-calculation timers are set, and how big the map is — all design and tuning choices, not a property printed on the box. If a network recovers slowly, the fix is almost never "swap the protocol".

Difference 1: how their messages travel

Think of it as two ways of sending the map updates. OSPF sends its messages through the normal postal system. Its notes ride inside IP — the same addressing system your data uses — on IP's own numbered channel 89 (engineers say "protocol 89"; IP has numbered channels the way a building has numbered doors, and OSPF has its own door). IS-IS uses its own private courier instead. Its messages travel directly over the cable itself, one hop at a time, without using IP at all. Its routers do not even introduce themselves with IP addresses: each one carries a name tag from an older addressing family, called a NET (Network Entity Title) — it looks like 49.0001.1921.6800.1001.00.

Why would anyone want the private courier? Because of the worst day of the year. If the IP addressing on a link is broken or mis-typed, OSPF's messages cannot travel that link — the very system that would fix the addressing problem is cut off by it. IS-IS messages do not care: the courier does not need street addresses to walk next door, so the neighbour partnership stays up and the network can still heal itself. The price of that independence is small but real: before anything else makes sense, you have to learn to read those name tags.

Difference 2: how they divide a big map

A map of ten routers fits on one page. A map of ten thousand does not — so both protocols let you cut the map into pieces, and each router only keeps the fine detail for its own piece. This is where their shapes genuinely differ.

OSPF works like city districts around one central station. Each piece of the map is an area. One special area, Area 0, is the backbone, and the rule is strict: every other area must connect directly to Area 0, and traffic between areas crosses the backbone. Done well, areas keep each router's map small — a router only needs street-level detail for its own district. Done carelessly on day one, they add border crossings a small network never needed.

IS-IS works like local streets feeding one long ring road. It divides by level instead of area number. Level 1 routers know their own local area in detail. Level 2 routers know how to travel between areas. A Level 1/2 router does both jobs. The backbone is not one special area — it is simply the unbroken chain of Level 2 and Level 1/2 routers linked together, wherever they happen to be. And here is the part big networks love: you are allowed to skip the local level almost entirely and run one huge, flat Level 2 map. Doing the same in OSPF means one giant Area 0, which its design tolerates far less gracefully.

Difference 3: how they grow new features

Remember the two filing systems from the start. This is where the labelled slips pay off. Almost everything IS-IS carries is a TLV — a slip that says "here is what I am, here is how long I am, here is the fact". When the world invented new facts to carry — IPv6 addresses, traffic-engineering numbers, the labels used by segment routing — IS-IS just added new kinds of slip. The folder never changed; the slips multiplied. One IS-IS setup therefore carries IPv4 and IPv6 together, in a single map.

OSPF's paperwork is more bound to its numbering system, so growth historically meant printing new kinds of form — and for IPv6, printing a whole second manual. IPv6 in OSPF is a separate sibling protocol, OSPFv3, usually run as a second, parallel setup next to OSPFv2: two neighbour partnerships, two maps, two of everything. (OSPFv3 can carry IPv4 as well, on paper — but the two-setup pattern is still what most engineers meet in the field.) At backbone scale, IS-IS's one setup carrying both families is a genuine saving in daily work, not a footnote.

The smaller print

Who chairs the meeting. When several routers share one cable (a typical office LAN), having everyone exchange notes with everyone is noise. So one router is elected chair: in OSPF it is called the designated router, with a backup chair (the BDR) ready to take over. IS-IS elects a chair too (the DIS) but keeps no backup — and if a stronger router joins later, it can take the chair away, which OSPF's chair never suffers. Small difference, slightly different behaviour when a chair fails.

Price tags that can go high enough. Remember each road gets a price tag, and the maths picks the cheapest route — so the tags must be able to tell a fast road from a slow one. Old IS-IS price tags stopped at 63, like a price list that cannot tell a footpath from a motorway; every modern IS-IS network switches to "wide" tags, which reach 16,777,215. OSPF has the mirror-image trap: it works each tag out from the link's speed against a reference speed, and with the factory reference, every modern 10G, 25G and 100G link rounds down to the same price of 1 — the maths then cannot prefer the motorway. Raising that reference is a one-line fix that old networks often never got.

Side by side

Why the giant backbones picked IS-IS

Three reasons, in the order the people who run them usually give. First, size without border gymnastics: one flat Level 2 map with hundreds of routers is a normal IS-IS design, so the backbone grows by bolting on routers, not by re-drawing district borders and re-planning which area touches the centre. Second, new features arrived cheaply: because labelled slips made growth easy, IS-IS picked up IPv6, traffic engineering and later segment routing cleanly. Segment routing, in plain words, means writing a packet's route as a short list of stops stuck to the packet itself — and the giant provider cores that use it very often run IS-IS underneath to hand out those stops. Third, it does not depend on the thing it fixes: the system that heals the network can still form its neighbour partnerships on a link whose IP addressing is broken — one less circular problem on the worst day of the year.

None of that makes IS-IS "better" in a 40-router campus. It makes it better fitted to one particular shape of problem: huge, long-lived, hungry for new features, and run by a small expert team who chose it once and have kept it for decades.

Why companies picked OSPF — and were right to

A company network has a different shape: many sites, many hands, people changing jobs, and real value in the next engineer understanding the design at 2 a.m. OSPF wins there on people, not packets. It is taught first in almost every course, it appears in every vendor's company-network design guide, every tool supports it, and the pool of engineers who can run it is the widest there is. Its district model, whatever its rules, also matches how companies actually look: a head-office backbone with branch districts around it. And for a normal estate, running OSPF is a solved, boring problem — and in operations, boring is a compliment.

One honest third answer belongs in any modern comparison. Inside many of today's data centres, the switches are arranged like a ladder: a bottom row plugged into the servers, a top row connecting the bottom row together (engineers call this a leaf-spine fabric). In those buildings, the plumbing routes underneath everything — the underlay — are often run by neither protocol. They use eBGP instead: the between-organisations protocol, borrowed inside, giving each switch its own number and simple one-to-one partnerships with its neighbours — no shared map to flood at all. So "which of the two should the new fabric run?" is no longer always a two-way question.

So: which should you learn, and which should you run?

Learn OSPF first if you are heading for company roles. It is the interview and certification baseline, and its map-and-shortest-path ideas carry straight over to everything else. Learn IS-IS next if you are aiming at carriers, cloud giants, high-frequency trading or segment-routing work — job adverts in those worlds name it, and courses rarely teach it, which is exactly why our full IS-IS guide (linked below) exists. Knowing both, and knowing they are cousins rather than rivals, is the real senior-engineer signal.

Run OSPF for a small-to-large company network with a mixed team and standard equipment — one area until the map genuinely grows big enough to need districts. Run IS-IS for a carrier core, a very large backbone, or a segment-routing underlay looked after by one expert team — wide price tags, mostly Level 2. For a brand-new leaf-spine data centre, look at eBGP for the plumbing before defaulting to either. And in every case: the protocol your on-call team can troubleshoot half-asleep beats the prettier one they cannot.

Four mistakes this comparison prevents

1. Copying a carrier design into a campus. A flat, all-Level-2 IS-IS core is elegant at 500 routers and pointless ceremony at 15, where a single-area OSPF design would be understood by every new hire on day one. 2. Splitting OSPF into areas on day one. Areas are a scaling tool, not a rite of passage. Splitting early means border routers (the ABRs that sit between areas) and summary lines — writing "all of the north side" on the map instead of listing every street — long before the map is big enough to need either. 3. Leaving factory price tags in place. IS-IS's old tags cannot tell a 100G link from a 1G link, and OSPF's factory reference speed flattens every modern link to price 1 — both quietly produce nonsense routes on fast networks until the wide tags or the raised reference are set. 4. Arguing about the protocol instead of checking the design. Slow settling, partnerships that keep dropping, and odd route choices are almost always timer settings, a mismatched cable setting, area design, or a physical fault first. Check the design before blaming the acronym.

Try it yourself

See an OSPF design mistake the way a checker sees it: paste a small multi-router config set into the free network config verifier and watch it flag an OSPF area mismatch between neighbours — the exact class of fault the area model creates. Then model what the winning protocol has to carry with the Clos fabric designer, and if segment routing is why IS-IS caught your eye, compute a real policy path in the Segment Routing Policy Lab.

Key takeaways

• OSPF and IS-IS do the same job the same basic way: every router shares what it can see, all of them build the same map of the network, and each one picks the shortest route over that map. Learn one well and you already understand most of the other.
• The biggest real difference is how the map is divided. OSPF splits the network into areas, and every area must connect to one central backbone area. IS-IS splits it into levels, and its backbone is simply a chain of routers — so a huge network can stay as one flat map with no central area to design around.
• IS-IS is also easier to extend: new features are added as new labelled notes inside its messages, and one IS-IS setup carries both IPv4 and IPv6 traffic. OSPF usually needs two separate setups, one for each.
• Which to pick is mostly about people, not speed: OSPF is the safe default for most companies because engineers, training and tools for it are everywhere. IS-IS fits giant, long-lived backbones run by a small expert team. Many modern data centres pick a third option — eBGP — instead of either.

Further reading

• IS-IS Explained: The Routing Protocol Carriers and Hyperscalers Actually Run
• BGP: How the Internet Actually Finds a Path
• Segment Routing Explained: How a List of Labels Replaces RSVP
• ECMP Explained: How Data Centers Spread Traffic Across Every Link

Fact-check sources: OSPFv2 — RFC 2328 (OSPF runs over IP, protocol 89; area model with backbone Area 0; DR/BDR election). OSPF for IPv6 — RFC 5340 (OSPFv3); OSPFv3 address families — RFC 5838. IS-IS for IP networks — RFC 1195 (integrated IS-IS; Level 1 / Level 2 routing; NET addressing); IS-IS runs over the link layer per ISO/IEC 10589. Wide metrics — RFC 3784 (24-bit metric TLVs, replacing narrow metrics capped at 63 per link). IPv6 in IS-IS — RFC 5308. Segment routing over IS-IS/OSPF extensions — RFC 8660 / RFC 8663 family. eBGP in large-scale data centres — RFC 7938. No vendor market-share figures or named-operator deployments are quoted; where each protocol is 'usually' found is stated as common field practice, not a measured statistic.

Packet Path is a free site for network engineers — 31 browser tools and plain-English guides, built with AI help on the build and drafting. The original version of this article is on the site: IS-IS vs OSPF: Which One Should You Learn — and Run?.

Why does an AI cluster care whether Ethernet drops packets?

Imagine a warehouse where forklifts drive loads straight into the right shelf slot in another building — no front desk, no paperwork, nobody checking them in. That is RDMA (Remote Direct Memory Access): one machine's network card puts data directly into another machine's memory. It is extremely fast precisely because almost nothing in software touches the data on the way.

The catch is built into that design. Software checks are what normally notice a lost packet and quietly ask for it again. An RDMA card expects the network to deliver — when a packet does drop, recovery is slow and coarse: a transfer can stall while a whole window of data is sent again, and a training job with thousands of GPUs all waiting on each other feels one machine's stall everywhere. Ordinary Ethernet makes no promises: when a switch queue fills, it drops. That is fine for web traffic. It is poison for RDMA.

In plain English, RoCEv2 is RDMA riding inside ordinary routed Ethernet packets (UDP destination port 4791, so any IP fabric can carry it). Because the Ethernet underneath still wants to drop packets under pressure, a RoCEv2 fabric adds three helpers to fake a network that never drops: PFC pauses senders instead of dropping, ECN warns senders early, and DCQCN is the rulebook that turns those warnings into smooth speed changes.

Helper 1: PFC — the emergency brake

PFC (Priority Flow Control, IEEE 802.1Qbb) changes what a full queue does. Instead of dropping the next RoCE packet, the switch sends a pause frame to the device directly upstream: stop sending this one traffic priority — the lossless one carrying RDMA — while other priorities keep flowing. When the queue drains, a resume lets traffic start again. Nothing drops, so RDMA never notices.

Two details make or break it:

• Pausing is not instant. Packets already on the wire keep arriving after the pause is sent, so every port needs headroom buffer — reserved space to catch those in-flight packets. Longer cables put more bits in flight, so longer links need more headroom; run out of headroom and you drop packets on the one class that must never drop.
• PFC works hop by hop. A congested receiver pauses its leaf, the leaf's buffers fill and it pauses the spine, and congestion can climb the fabric tree — helpful for a brief burst, dangerous when it lasts.

Note the price: the sender fully stops. It does not slow down.

Helper 2: ECN — the early warning

Pausing is a blunt tool: a sender is either full speed or fully stopped. ECN (Explicit Congestion Notification) is the gentle tool. When a switch queue passes a marking threshold — starting to build, nowhere near full — the switch sets a mark in the IP header of passing RoCE packets instead of dropping or pausing anything. The mark rides the packet to the receiver, whose card answers with a small CNP (Congestion Notification Packet): the path is getting busy, ease off.

Think of the difference like traffic control on a highway. PFC is a roadblock: everything in one lane stops. ECN is the brake-light wave and the variable speed-limit sign: everyone slows a little, early, and the roadblock never becomes necessary. A healthy RoCE fabric lives almost entirely on ECN signals. PFC exists for the moments ECN could not prevent — a sudden burst arriving faster than feedback can travel.

Helper 3: DCQCN — deciding how much to slow down

A warning is useless without a rule for reacting to it. DCQCN (Data Center Quantized Congestion Control) is that rule, running in the network cards. When CNPs arrive, the sender cuts its rate — by an amount that grows if marks keep coming. When marks stop, the sender cautiously probes upward again, in small steps, until it finds the rate the fabric can actually sustain.

Thousands of senders doing this at once is what keeps an AI fabric busy but not clogged: queues stay short, PFC stays silent, and collective operations — all GPUs swapping results at the same moment, the normal rhythm of training — finish on time.

The tuning tension: ECN must bite before PFC fires

Here is the one idea to keep if you keep nothing else. Both mechanisms watch the same queue. ECN marking is set by a queue threshold; PFC pause is set by a higher buffer threshold (plus that reserved headroom). For the system to work, the order on the way up must be:

queue starts building → ECN marks → CNPs fly → DCQCN slows senders → queue drains → PFC never fires.

Get the order wrong and the fabric misbehaves in predictable ways. ECN thresholds set too high (or PFC thresholds too low) mean pauses fire before feedback can help: senders lurch between full speed and full stop, queues elsewhere sit empty, and training throughput sags. ECN thresholds set too low mean senders are told to slow down for queues that would have drained on their own: the fabric runs permanently below capacity.

And because every vendor exposes these knobs differently — queue thresholds in bytes or cells, marking probabilities, CNP rates, DCQCN timers — values copied from another site's runbook are a starting point to validate under load, never a guarantee.

When lossless goes wrong: storms and deadlock

PFC's hop-by-hop kindness has two famous failure modes:

• A PFC storm is a device that pauses and never resumes — one sick card can freeze a priority across every switch it can reach, which is why fabrics run PFC watchdogs that intervene when a port stays paused abnormally long.
• PFC deadlock is subtler: imagine a ring of switches, every buffer full, each switch paused waiting for the next one downstream to resume — and the next one is waiting on it. Nobody can drain, so nobody resumes. It needs a cycle in buffer dependencies, which is why lossless classes are kept few (usually one) and topologies and routing are designed to avoid those cycles.

One more everyday hazard: incast. AI collectives make dozens of senders transmit to one receiver at the same instant by design. That receiver's queue is where your ECN thresholds, headroom, and DCQCN settings stop being theory — size and tune for the collective pattern you actually run, and spread flows with good hashing.

Five mistakes that break RoCE fabrics

1. Turning PFC on for everything. Every lossless priority multiplies headroom use and backpressure spread. Keep lossless to the RDMA class; let ordinary traffic stay lossy — TCP handles drops fine, that is what it was built for.
2. A marking mismatch somewhere in the path. The DSCP-to-priority mapping, ECN marking, and PFC class must agree on the NICs and every switch hop. One hop that maps RoCE to a different priority silently turns that link lossy — and RDMA finds out first.
3. ECN thresholds above the PFC trigger. That inverts the intended order: the fabric pauses before it ever warns. Check the effective queue depths, not just the numbers typed into two different config screens.
4. Ignoring cable length in headroom. Headroom must cover the bits in flight while a pause travels and takes effect. Swap in longer cables or faster links without revisiting headroom and the lossless class starts dropping.
5. Tuning on a quiet fabric. All three mechanisms only act under congestion. Validate with a real collective or incast pattern at production scale — a fabric that looks perfect at 10% load has told you almost nothing.

Try it yourself

Build the fabric intuition this runs on with free tools: size the spine-leaf fabric itself with the data-center capacity planner, watch how flow hashing spreads — or fails to spread — collective traffic in the ECMP hash polarization visualizer, and check the MTU side of a jumbo-frame RoCE design with the MTU/MSS tunnel overhead calculator.

Key takeaways

• RoCEv2 is RDMA over routed UDP/IP (destination port 4791): the network card moves data straight into another machine's memory, bypassing the CPU and the kernel — which is why it is fast, and why it assumes packets will not drop.
• PFC is the emergency brake: a congested switch pauses one traffic priority at a time instead of dropping its packets. It prevents loss, but a pause can spread congestion upstream and, in a loop of full buffers, deadlock the fabric.
• ECN is the early warning: switches mark packets when a queue starts building, the receiver answers with a CNP, and DCQCN cuts the sender's rate smoothly — so traffic slows down before PFC ever has to fire.
• The whole tuning game is ordering: ECN marking thresholds must bite below the queue depth that triggers PFC.

Further reading

• Why AI Clusters Have Two Networks — the front-end vs back-end split; RoCEv2 lives on the back-end fabric.
• ECMP Explained — how fabrics spread flows across links, and the polarization trap that concentrates them instead.

Packet Path is a free site for network engineers — 28 browser tools and plain-English guides, built with AI help on the build and drafting. The original version of this article is on the site: RoCEv2 Explained: PFC, ECN & DCQCN for Lossless AI Fabrics.

One best path is a wasted fabric

Picture a modern spine-leaf data center: every leaf switch connects to every spine. Between any two racks there aren't one or two paths — there are dozens, all the same length, all the same cost. Now ask the router how it forwards. Classic routing picks exactly one best path and programs it into the forwarding table. The other thirty-one links become expensive decoration. You've built a 32-lane highway and you're driving every car down a single lane.

Equal-cost multi-path routing — ECMP — is the fix. When the routing table holds several equal-cost routes to the same destination, the router installs all of them and spreads traffic across every link. In hyperscale fabrics ECMP isn't an optimization; it is the forwarding model. And understanding its failure modes is the difference between a fabric that hums and one that mysteriously congests at 40% utilization.

What ECMP actually does

It starts in the control plane. OSPF, IS-IS, or BGP computes routes; when several routes to a prefix tie — same OSPF cost, or BGP multipath policy — the RIB keeps all of them instead of one winner. The forwarding plane programs an ECMP group (vendors call it a next-hop group) into the FIB: one destination, N next-hops, N egress interfaces.

Each arriving packet is assigned to one member of the group. The assignment happens in hardware, per packet, at line rate — and it must be consistent: every packet of a flow has to take the same path, or TCP sees reordering, assumes loss, and collapses its congestion window. So the switch doesn't round-robin packets. It hashes flows.

The hash: five fields in, one link out

The classic ECMP hash input is the 5-tuple: source IP, destination IP, protocol, source port, destination port. Platforms vary — some mix in the ingress port or VLAN ID. The hash output picks the egress member, conceptually member = hash(fields) mod N.

Two properties carry the whole design. Determinism: the same 5-tuple always selects the same member, so flows never reorder. Distribution: across many flows, each member should receive roughly 1/N of them.

Burn this distinction in: ECMP balances flows, not bytes. Ten thousand small flows spread beautifully. Three elephant flows can land on the same member and saturate it while sibling links idle. "Even hash distribution" and "even link utilization" are not the same thing — and confusing them is the root of half of all ECMP troubleshooting sessions.

Hash polarization: the silent killer

Now the failure mode that burns experienced engineers. The hash is deterministic — and on many platforms it is the same deterministic function, with the same seed, on every switch. Follow one flow across three hops: the leaf hashes its 5-tuple and picks member 2; the spine hashes the identical 5-tuple with the identical function and picks its member 2; the next leaf does it again. Not just this flow — every flow with a similar 5-tuple rides the same relative link at every tier. Traffic polarizes onto a handful of links instead of spreading across the fabric. You get hot spots, drops, and retransmissions while most of the fabric sits idle — and nothing in any routing table looks wrong.

Polarization is vicious because every device is behaving correctly in isolation. The bug is emergent: identical hash behavior everywhere. It presents as persistent, unexplained congestion on a few links in an otherwise healthy fabric, and it survives reboots, because determinism is the point.

The rule: Polarization is a property of the fabric, not of any single device. If every switch hashes identically, your 32-way ECMP is effectively 1-way for correlated flows.

Breaking the polarization

Every fix introduces per-device variation into the hash:

• Hash seed or offset per device. Most vendors expose a configurable hash seed. Different seed, different member selection for the same flow, polarization broken. This is the first knob to reach for, and the one most often left at default.
• Vary the hash inputs by tier. Hash on different field combinations at different layers — include the ingress port at the leaf, exclude it at the spine. Same flow, different inputs, different path.
• Resilient hashing. Modern implementations remap only the flows hashed to a member that disappears, instead of rehashing the entire table on every link flap. It matters during maintenance windows and brownouts.

See it with your own eyes: the free ECMP Hash Polarization Visualizer simulates the hash hop by hop across a fabric. Toggle identical versus seeded hash functions and watch flows collapse onto the same links — or spread the way you intended.

ECMP in the wild: protocols and overlays

In the underlay, OSPF and IS-IS give you ECMP almost for free: equal-cost paths are installed by default up to the platform's max-paths limit — check it, because 8, 16, 32, and 64 are all common ceilings, and silently exceeding one is a classic. BGP needs explicit multipath configuration, and at hyperscale the interesting knobs are multipath relax and Add-Paths for backup diversity.

Overlays add a wrinkle that bites everyone once: VXLAN. The underlay ECMP hash sees only the outer headers — outer source and destination IP, outer UDP ports. If the encapsulation stamps the same outer source port on every inner flow, the underlay sees one giant flow and pins it to a single link. The fix is UDP source-port entropy (RFC 7348): the VTEP hashes the inner 5-tuple into the outer source port, restoring per-flow distribution to the underlay. If your VXLAN fabric shows one hot spine link, check entropy before you check anything else.

Troubleshooting lopsided links

When link utilization is uneven and the routing table is innocent, work this list:

• Count flows per member, not just bytes. A member carrying 3 elephant flows at 90% while others carry 3,000 mice at 10% is a distribution problem, not a capacity problem.
• Check max-paths. If the platform installed 8 members but 16 equal paths exist, you're blind to half your fabric.
• Compare hash seeds across tiers. Identical seeds on leaf and spine is polarization waiting to happen — or already happening.
• Verify VXLAN entropy. One outer source port for all inner flows is the overlay version of the same disease.
• Look for asymmetric hashing on bundled links. LAG and ECMP hashing interact; a flow pinned by ECMP to one member can be pinned again inside a port-channel.

Where this fits

ECMP is the forwarding half of the hyperscaler story; BGP is the control-plane half. Read them together: BGP: How the Internet Actually Finds a Path covers how those equal-cost routes get computed in the first place.

Further reading:

• ECMP Hash Polarization Visualizer — watch polarization happen hop by hop, then fix it with hash seeds.
• TCP Internals Every Network Engineer Should Know — why reordering from inconsistent paths destroys throughput.
• Packet Path — more free tools and plain-English guides for network engineers.