Batch inference scores many rows on a schedule and stores the predictions for later lookup; online inference scores each request on demand, within a latency budget. Batch is cheaper and simpler, while online can use features from the moment of the request.
Card 1 of 228. Answer: Batch inference scores many rows on a schedule and stores the predictions for later lookup; online inference scores each request on demand, within a latency budget. Batch is cheaper and simpler, while online can use features from the moment of the request.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
In batch inference a job runs, say, nightly, scores every user or item, and writes the predictions to a table; the product reads them with a lookup. In online (real-time) inference the model sits behind an API and scores each request as it comes in, typically within tens of milliseconds. It is a separate question from online versus batch learning, which is about how the model is trained.
Example: a shop precomputes the top 20 products for each of its 5 million users every night. Serving is a fast key-value read. But a user who signed up an hour ago has no row, and nobody's recommendations reflect what they clicked five minutes ago. Scoring the session online fixes both, at the cost of a service that must return in time and features that must be available in milliseconds.
Choose by how fast the inputs change and how long the product can wait. Batch wastes compute on users who never return and serves stale predictions; online needs low-latency feature serving and makes training-serving skew easier to introduce. Many systems mix the two: batch jobs generate candidates or embeddings, and an online model re-ranks them with fresh context.
