Research-backed calculator guide

Vertex AI Online vs Batch Prediction Cost: Choose the Right Serving Mode

Compare Vertex AI online and batch prediction cost with formulas, a worked node-hour example, endpoint and storage assumptions, and a practical decision framework.

What Vertex AI online versus batch prediction cost actually measures

Online prediction: pay for readiness as well as work

An online endpoint exists to accept a request at serving time. The endpoint must have a deployed model and compute resources associated with it. That readiness is the key cost behavior: a lightly used endpoint can still consume node-hours while it is available. Google’s pricing documentation says that a deployed model is charged even when no prediction is made, and that you must undeploy it to stop further deployment charges. The same page distinguishes the online priority of reducing individual-request latency from batch’s priority of reducing total job elapsed time. ^[2]

For a first-pass online estimate, use:

online compute = average online nodes × active hours × price per node-hour

That formula is intentionally simple. Replace “average online nodes” with a measured average if autoscaling changes capacity during the month. Add accelerator hours when the deployment uses them. If the endpoint has multiple deployed models, multiple traffic splits, or separate staging and production resources, model those resources separately rather than treating the endpoint as one request meter.

Batch prediction: pay for finite replica runtime

Batch inference starts with a known input set and a requested replica count. The service partitions the work and writes the predictions to Cloud Storage or BigQuery. For custom trained models, Google notes that batch jobs do not autoscale in the same way as online inference; the job uses the starting replica count, and the maximum replica count is ignored. ^[3]

For a first-pass batch estimate, use:

batch compute = replicas × billed job hours × price per node-hour

The billed job hours must include startup and teardown behavior that falls inside the provider’s billable runtime. Google’s guidance recommends targeting at least about 10 minutes of useful batch runtime because each replica includes approximately five minutes of startup time in the billed node-hour. GPU machines can take longer to start, so a very short GPU job can waste a larger share of its budget before producing predictions. ^[3]

This is why a batch job with fewer records is not automatically free. A job that processes only a few seconds of data may still incur a meaningful minimum of replica runtime. Conversely, increasing replicas can reduce elapsed time without reducing the total node-hours proportionally. Benchmark the actual model, batch size, machine type, and input format before committing to a production schedule.

MaaS and generative-model caveat

“Vertex AI batch pricing” is not one universal meter. Google’s MaaS batch-prediction documentation lists supported models and expects JSONL or BigQuery inputs for that capability. ^[4] A generative model can also have model-specific input and output token charges rather than the custom-model machine-rate pattern used in the worked example. Check the specific model’s current pricing and eligibility before applying a node-hour worksheet. The example below is for comparing infrastructure utilization around a custom-model-style deployment; it is not a quote for every Gemini, partner, embedding, or Model Garden workload.

Worked cost example

Assume the following scenario: a fraud model scores a file once per week, can wait 25 minutes for results, and has no interactive request requirement. The same model could instead sit behind an online endpoint for an application that needs a response immediately. We use one n1-standard-4 node in us-central1 and the on-demand hourly rate shown on Google’s checked pricing page. The figures are rounded only for display.

Under these assumptions, the endpoint’s compute subtotal is 432 times the four-batch compute subtotal. That ratio is not a general Vertex AI promise. It is a utilization result: 720 online hours divided by four 25-minute jobs equals 432. If online traffic requires three nodes at peak, or if batch needs ten replicas for a shorter deadline, the result changes immediately.

The table also shows what it does not include. Add Cloud Storage charges for input and output objects, BigQuery storage or query charges when BigQuery is used, model artifact storage, network egress where applicable, accelerator rates, monitoring, orchestration, retries, and any model-specific token or request charges. Google requires batch input and output locations to be aligned with the model’s region or multi-region rules, so a design that ignores data placement can produce a misleading compute-only comparison. ^[3]

A practical monthly worksheet can therefore use two columns:

1. Serving compute: online average nodes × hours, or batch replicas × billed runtime × jobs.
2. Pipeline and product layers: storage, BigQuery, network, orchestration, monitoring, retries, and the value of faster results.

The second column is often small for a tiny file and material for a large recurring pipeline. Do not hide it inside a guessed “markup.” Record each assumption so your next invoice can be reconciled against the estimate.

How to choose the cheaper feasible mode

Start with the latency constraint, not the price table. If the caller needs a fraud decision during checkout, a recommendation before a page renders, or a moderation response before a post is published, batch is not a feasible substitute merely because its compute window looks cheaper. Online inference is designed for timely application input, while batch is designed for accumulated data and deferred results. ^[1]

Next, confirm model eligibility. An online design needs a model that can be deployed to an endpoint and a serving configuration that meets the latency and concurrency requirement. A batch design needs an eligible batch path, a supported input format, and a destination. Google’s MaaS documentation specifically says to prepare JSONL in Cloud Storage or a BigQuery table and allows output to either BigQuery or Cloud Storage for that capability. ^[4]

Then benchmark the data path. Cloud Storage JSONL, CSV, file lists, TFRecord, and BigQuery do not have identical parsing and partitioning behavior. Google’s custom-model guidance says BigQuery, file-list, and JSONL inputs are automatically partitioned, while CSV is not naturally partition-friendly for throughput-sensitive applications. ^[3] A lower machine price can be erased by poor partitioning, oversized requests, retries, or a region mismatch.

Finally, compare the operating schedule. An online endpoint can be economical when it is serving continuously and its readiness is necessary. For development or occasional scoring, an always-ready endpoint can become a recurring idle charge. A batch job can be economical for a nightly or weekly workload, but it still needs job monitoring, failure handling, output retention, and a plan for late or incomplete results. If the workload moves from four scheduled jobs to near-continuous traffic, recompute the model rather than assuming the original conclusion remains valid.

For adjacent planning, the Model pricing comparison guide helps normalize provider and model meters, while the RAG, embedding and vector database cost calculator is useful when predictions are only one stage in a retrieval pipeline. If the serving path is part of a larger build, use the AI agent development cost estimator to keep implementation and run-cost assumptions separate.

Common exclusions and limitations

Storage is not automatically zero. Batch inputs and outputs commonly live in Cloud Storage or BigQuery. The batch job may be cheap while the retained datasets, intermediate files, and result tables accumulate separate charges. Google’s BigQuery workflow requires an input table with the expected request structure and a destination table or Cloud Storage output, with the job and table in the same region. ^[5]

A node-hour is not a prediction count. Two jobs with the same record count can have different costs if model execution time, batch size, machine type, accelerators, or retries differ. Measure records per second and billed runtime from a representative sample.

The pricing page can change. This article records a US-dollar, us-central1 assumption checked on 2026-11-02. Region, machine family, consumption plan, model generation, and service naming can change. Recheck the current provider page before using the number in a budget or purchase decision. Discounts shown on the provider page are not assumed here.

A compute-only break-even is not a business break-even. Online serving can justify its extra capacity when a delayed decision has a measurable product or operational cost. Batch can be the right answer even when its bill is not the absolute minimum if it simplifies governance, reproducibility, and bulk reprocessing. Conversely, batch can be wrong when freshness, p95 latency, or failure recovery has a hard service-level requirement.

FAQ

Is Vertex AI batch prediction always cheaper than online prediction?

No. Batch often avoids paying for an endpoint that must remain ready between requests, but it can use many replicas, incur startup time, require more retries, or add BigQuery and storage work. Online can be reasonable for sustained utilization. Compare measured billed node-hours and all required data services for the same workload and deadline.

Does online prediction charge when there are no requests?

For deployed custom or AutoML-style models, Google’s pricing documentation says you pay for a model deployed to an endpoint even if no prediction is made, and you must undeploy it to stop further charges. ^[2] The exact behavior depends on the product and deployment type, so confirm the current pricing page for the model you use. Treat an idle endpoint as a capacity cost in your forecast rather than assuming request volume is the only meter.

Do batch prediction jobs need an endpoint?

For the standard batch-inference workflow, Google describes batch as an asynchronous request made directly to a model that is not deployed to an endpoint. Online inference, by contrast, requires the model to be deployed to an endpoint. ^[1] Some products and model families have their own eligibility rules, so check the relevant batch documentation rather than transferring this rule to every Vertex AI feature.

How do I calculate Vertex AI batch inference cost?

Start with replicas × billed hours × applicable node or machine rate. Include startup time, accelerator hours, and the number of scheduled jobs. Then add Cloud Storage, BigQuery, network, monitoring, orchestration, retries, and any model-specific token or request charges. Run a small representative benchmark because Google’s replica and throughput guidance is approximate and depends on the model and input format. ^[3]

When should I use BigQuery instead of Cloud Storage for batch input?

Use the source that matches where the data already lives and how downstream users consume results. BigQuery can avoid a separate export and supports table-to-table workflows, but it introduces dataset location, schema, permissions, storage, and query considerations. Cloud Storage JSONL can be simpler for file-oriented pipelines. In either case, keep the model, job, input, and output locations aligned with the provider’s regional requirements. ^[5]

Does the online-versus-batch answer apply to Vertex AI MaaS models?

Only after checking the model-specific documentation. MaaS batch prediction has a supported-model list and documented JSONL or BigQuery input requirements. ^[4] Generative models may also bill input and output tokens, and their batch and online rates may not match a custom machine-node example. Use the model’s current pricing page and preserve the distinction between token charges and serving infrastructure charges.