Research-backed calculator guide
Multi-GPU Cloud Networking Cost: Price Scaling Efficiency, Not Just GPU Hours
Compare multi-GPU cloud cost by separating GPU hours, network charges, and useful scaling efficiency with a worked, provider-neutral example.
Quick answer
A multi-GPU cloud networking cost comparison should not rank offers by GPU-hour price alone. Calculate three layers separately: the billed accelerator and instance cost, explicit data-transfer or networking services, and the useful work delivered after communication overhead. A simple model is:
Total run cost = (GPU rate × GPU count × billed hours) + network services + data transfer + storage/control-plane items.
Then divide by measured useful output, not by nominal GPU count. If an eight-GPU job achieves 72% scaling efficiency, its useful capacity is 8 × 0.72 = 5.76 ideal-GPU equivalents. A provider with a lower GPU rate can still lose on cost per useful result if its interconnect makes synchronization slow, causes idle time, or requires more orchestration.
This is an estimate framework, not a vendor quote. Cloud bills differ by region, instance family, commitment, operating system, network path, storage tier, minimums, and negotiated discounts. On 2026-11-02, Google’s Compute Engine pricing documentation stated that prices are listed in USD and that network usage is measured using GB/GiB conventions; it also separates accelerator-optimized machine families from other VM categories. [1] AWS similarly describes EC2 On-Demand pricing as instance-hour billing, with partial instance-hours billed per second for several operating systems, and treats data transfer as a separate pricing area. [4]
Why network-adjusted cost is the useful comparison
The network can affect your bill in two different ways. First, it can create explicit charges: inter-region transfer, internet egress, interconnect connections, VLAN attachments, load balancers, NAT, public IPs, or a managed cluster control plane. Second, it can reduce the amount of work completed per billed hour even when the network line on the invoice is zero. Distributed training and some large-model inference layouts repeatedly exchange gradients, activations, parameters, or requests. Waiting for those transfers is an economic cost because GPUs remain rented while they are not producing useful output.
Do not confuse the backend GPU fabric with ordinary application egress. A provider may bundle a high-speed fabric into a fixed multi-GPU instance, while charging separately for traffic leaving the region. Another provider may let you assemble independent machines and expose more line items. The comparison must document what each quote includes.
NVIDIA’s NCCL documentation exposes controls for peer-to-peer communication, InfiniBand, network GPUDirect RDMA, topology, and network selection. [5] Those controls are a reminder that “eight GPUs” is not a complete performance specification. Record the topology, interconnect class, bandwidth, latency, and the framework’s measured collective performance alongside the price.
A practical comparison therefore uses four numbers:
- Raw cluster rate: GPU or instance dollars per hour multiplied by the device count.
- Explicit network cost: recurring network resources plus data transferred over billable paths.
- Scaling efficiency: useful throughput divided by ideal linear throughput for the same device count.
- Effective cost: total dollars divided by useful cluster-hours, tokens, samples, or completed jobs.
The last number is the decision metric. Keep the first three visible so another engineer can audit the result.
A provider-neutral comparison method
Start with one workload and one measurement boundary. For inference, choose a model, precision, input/output token mix, concurrency, batch policy, and target latency percentile. For training, choose the dataset slice, global batch, precision, checkpoint frequency, optimizer, and number of steps. Do not compare a single-GPU throughput benchmark against an eight-GPU training run and call the difference networking cost.
Next, normalize units. Use dollars per GPU-hour or dollars per instance-hour for compute, dollars per GiB or GB for transfer, and dollars per month or hour for persistent resources. Google’s network pricing documentation distinguishes inbound transfer, VM-to-VM traffic, inter-region traffic, and internet transfer; it also shows that a region and a zone are not interchangeable billing concepts. [3] Google Cloud Interconnect pricing separately lists hourly connection and VLAN attachment charges, then applies data-transfer-out rates by geography and GiB. [2]
Then measure the job at one GPU, two, four, and eight GPUs if the workload permits. Define scaling efficiency as:
scaling efficiency = single-GPU throughput × GPU count ÷ measured multi-GPU throughput
If the single-GPU baseline is 1,000 samples per hour and eight GPUs produce 5,760 samples per hour, efficiency is 5,760 ÷ (1,000 × 8) = 72%. For a latency-sensitive inference service, substitute completed requests under the chosen latency target. A raw throughput number that violates the SLO is not useful capacity.
Finally, separate vendor quotes from your estimates. A provider’s published rate is a quote or list-price observation at the stated date. Your measured efficiency, transfer volume, failure allowance, and utilization are workload assumptions. Label them as assumptions and rerun the model when the workload changes.
Worked cost example
Assume a six-hour distributed run repeated 30 times in a 30-day month. The hypothetical quote is $4.50 per GPU-hour for eight GPUs. The workload generates 1,200 GiB of billable transfer at an illustrative $0.02 per GiB and uses a $0.10-per-hour network attachment for 720 hours. These network values are not a universal multi-GPU rate. They are chosen to show the arithmetic using published Google Interconnect units and must be replaced with the selected provider’s quote. Google’s documentation lists $0.10 per hour for several VLAN attachment capacities and gives geography-dependent transfer rates, including $0.02 per GiB for specified North American paths. [2]
The benchmark measured 72% scaling efficiency. The input/output table makes every assumption inspectable:
| Step | Input or assumption | Formula | Output |
|---|---|---|---|
| GPU compute | 8 GPUs; $4.50/GPU-hour; 180 billed hours | 8 × 4.50 × 180 | $6,480.00 |
| Transfer | 1,200 GiB; illustrative $0.02/GiB | 1,200 × 0.02 | $24.00 |
| Network attachment | $0.10/hour; 720 hours | 0.10 × 720 | $72.00 |
| Total estimated run-month | Compute + transfer + attachment | 6,480 + 24 + 72 | $6,576.00 |
| Useful capacity | 180 nominal cluster-hours; 72% efficiency | 180 × 0.72 | 129.6 useful 8-GPU-equivalent hours |
| Network-adjusted useful cost | Total ÷ useful capacity | 6,576 ÷ 129.6 | $50.74/useful cluster-hour |
The raw compute-only figure is $36.00 per nominal eight-GPU hour, because 8 × $4.50 = $36. The network-adjusted figure is not $36 divided by eight. It is $6,576 divided by the useful 129.6 cluster-hours delivered after scaling loss. The result is approximately $50.74 per useful eight-GPU-equivalent hour.
Now test the decision sensitivity. If a second provider charges $4.00 per GPU-hour but measures only 55% efficiency, compute is $5,760 for the same 180 hours. Holding the illustrative network items constant, total cost is $5,856. Useful capacity is 99 cluster-hours, so effective cost is about $59.15 per useful cluster-hour. The cheaper headline GPU rate loses under this assumption. That is not proof that one provider is better; it shows why the efficiency measurement belongs in the comparison.
If the second provider reaches 80% efficiency, its useful capacity is 144 cluster-hours and effective cost falls to about $40.67. The same price sheet can produce very different economics depending on topology and workload behavior. Use a confidence range when efficiency is uncertain. For example, report the 55%, 72%, and 80% cases rather than presenting one benchmark run as a permanent fact.
The example excludes taxes, storage, checkpoint reads and writes, CPUs, orchestration, support, failure retries, reserved-capacity discounts, spot interruptions, and idle time outside the six-hour run. Google notes that network prices can depend on whether traffic is within a zone, between zones, across regions, or through an interconnect. [3] AWS likewise lists data transfer as a separate pricing area and says that On-Demand instance charges run from launch until termination or stop, subject to the stated billing granularity. [4] Add those items when they are present in the quote.
What to request from a cloud provider
Ask for a written configuration, not only “H100 at $X per hour.” Request the exact GPU model and memory, GPU count per node, number of nodes, CPU and RAM, local and attached storage, interconnect technology, topology, advertised bandwidth, and whether the fabric is included. Ask whether the rate is per GPU, per VM, per node, or for the complete cluster.
For networking, request the region and zones, ingress and egress rates, inter-region rates, interconnect or VLAN fees, public IP and load-balancer charges, NAT or firewall processing, and minimum billing periods. Ask whether transfers between GPUs, nodes, storage, and control-plane services are billed differently. A quote that says “free egress” may still exclude storage requests, cross-region traffic, or a managed orchestration layer.
For performance, request a reproducible benchmark or run your own. Capture throughput, p50 and p95 latency where relevant, all-reduce or collective bandwidth for training, job-start time, interruption rate, and usable hours. A high advertised link speed is not the same as application-level goodput. Record software versions and NCCL or framework settings so a later run can explain a change.
For finance, request on-demand, committed, and interruptible prices separately. State whether the workload can tolerate preemption. A discount that causes repeated restarts can increase total dollars per completed job. Also ask how idle instances, failed provisioning, requeued jobs, and attached storage are billed.
Run your scenario
Turn this framework into an editable estimate.
Use dated, source-linked assumptions and see cost per request, successful task and month.
When a multi-GPU setup is worth the premium
Paying for a tightly coupled cluster can be rational when the workload communicates frequently, the model cannot fit on one device, or the deadline makes throughput more valuable than the lowest nominal rate. It is harder to justify when the workload is embarrassingly parallel, each task is independent, and a queue can spread jobs across separate GPUs with little synchronization.
For inference, compare the cost per successful request under the target latency and availability policy. For retrieval-heavy systems, network and storage movement may dominate the GPU line, so document fetch and vector-database placement matter as much as inter-GPU bandwidth. The RAG, embedding and vector database cost calculator is a useful companion when the model is only one part of the serving path.
For an API-backed design, model spend and infrastructure spend should be kept separate. The Model pricing comparison helps with model-rate normalization, while this guide focuses on the economics of rented accelerator capacity and its network-adjusted output. If engineering labor, orchestration, or migration is material, include the AI agent development cost estimator rather than hiding those costs in a vague overhead percentage.
FAQ
Is networking always an additional line item for multi-GPU cloud instances?
No. Some providers bundle the intra-cluster fabric into a multi-GPU instance or managed cluster price. Other designs expose separate interconnect, VLAN, transfer, load-balancer, or egress charges. Even when the invoice has no network line, networking can still affect useful cost through synchronization delay, idle GPUs, retries, or lower scaling efficiency.
What is the simplest formula for multi-GPU cloud cost?
Start with (rate × device count × billed hours) + explicit network and data-transfer charges. Then divide by measured useful output or by nominal cluster-hours × scaling efficiency. Keep storage, CPU, control plane, support, taxes, and failure retries as separate rows so the model is not mistaken for a complete quote.
Should I compare cost per GPU-hour or cost per useful result?
Use GPU-hour as an input and cost per useful result as the decision metric. GPU-hour is easy to publish but says nothing about communication overhead, utilization, job failures, or whether the measured throughput meets the latency target. Cost per useful sample, token, request, training step, or completed job is harder to fake and easier to audit.
How much scaling efficiency should an eight-GPU job assume?
Do not use a universal percentage. Measure it on the exact model, precision, batch, framework, topology, and communication pattern. A 72% figure in the worked example is an explicit assumption, not a market benchmark. Report a sensitivity range when the workload is still being designed.
Do I need to count data transfer between GPUs as egress?
Not automatically. GPU-to-GPU traffic may use a bundled local fabric, a priced VM network, an inter-zone path, or another provider-specific service. Ask which path carries the traffic and which project or resource receives the charge. Treat the provider’s billing definition as authoritative for the quote, then keep application-level measurements separate from invoice categories.
What should I do if a provider publishes no network price?
Mark that component as unknown rather than setting it to zero. Request a quote for the exact topology and region, or build a low/base/high sensitivity case. A comparison can still proceed, but its conclusion should state that missing pricing is an uncertainty and should not be presented as a confirmed saving.
Limitations and assumptions
This guide uses USD and an English, United States research locale. Pricing observations and product descriptions were checked on 2026-11-02 and can change. Provider list prices are not negotiated quotes. The worked example uses hypothetical GPU pricing, transfer volume, attachment hours, and measured efficiency to demonstrate a method; replace them with your invoice and benchmark data.
The method does not infer search volume, ranking difficulty, traffic, impressions, revenue, or provider quality. It also does not claim that one cloud is cheapest. Network topology, region, workload, software stack, capacity availability, commitment terms, and failure behavior can reverse a comparison. Exclude any unavailable component from the headline total, label it unknown, and show a sensitivity range instead of inventing precision.
Primary sources checked
Pricing and product behavior can change. These sources support the time-sensitive claims in this guide; verify current provider terms before committing spend.