Training a machine learning model is a batch job that runs for hours or days. Serving predictions from that model to users requires sub-100ms latency. These two requirements produce completely different infrastructure. The model serving layer is where the offline ML world meets the online serving world.

The Prediction Service#

A model serving service exposes an API: input features in, prediction out.

POST /predict
{
  "model": "fraud_detection_v3",
  "features": {
    "amount": 1250.00,
    "merchant_category": "electronics",
    "user_avg_transaction": 85.00,
    "time_since_last_transaction_seconds": 3600
  }
}

→ { "prediction": "low_risk", "score": 0.08 }

The model is loaded into memory on service startup. Inference is purely computational: matrix multiplications (for neural networks) or tree traversals (for gradient boosting). No database calls, no external service calls. A well-implemented model serving layer can return predictions in single-digit milliseconds.

Feature Serving#

The hard part isn’t the model: it’s getting the features. The API above requires four input features. In a real fraud detection system, there are hundreds. Some are easily available at request time (the transaction amount), some require database lookups (user’s 30-day average transaction amount), and some require joining multiple sources (device fingerprint from session service, merchant risk score from merchant service).

A feature store pre-computes the features that can be computed in advance. At prediction time, the serving layer fetches pre-computed features from the online feature store (Redis or equivalent, sub-millisecond lookups) and combines them with the request-time features.

graph TD A[Prediction request: transaction data] --> B[Feature Assembly Layer] B --> C[Request-time features: amount, merchant, timestamp] B --> D[Online feature store: user history, device risk] C --> E[Complete feature vector] D --> E E --> F[Model inference: matrix multiply or tree traversal] F --> G[Prediction: low_risk, score 0.08] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Hardware: GPU vs CPU for Inference#

Large neural networks (recommendation models, large language models) benefit from GPU inference: the matrix multiplications that dominate inference time are parallelizable across GPU cores. A GPU can compute a prediction 10-100x faster than a CPU for large models.

For smaller models (gradient boosting for fraud detection, logistic regression for ad click prediction), CPUs are more cost-effective. The model is small enough that it fits in CPU cache, and memory bandwidth matters more than parallelism.

Model Loading and Memory#

A model file for a large recommendation system can be gigabytes. Loading it on every request is not feasible. Models are loaded once at service startup and held in memory. Multiple versions of a model might be in memory simultaneously during a model update, before the old version is evicted.

Memory pressure from large models is real. A service that holds 5 model versions (for gradual rollout) uses 5x the memory of one that holds 1. Memory management for model serving is a first-class concern.

At Oracle#

We didn’t run ML models in production, but we had configuration scoring: a rule engine evaluated 200+ conditions against network configs to assign a “compliance score.” This was conceptually identical to model inference: take a feature vector (config attributes), run it through a scoring function (rules instead of a model), return a score. The performance constraints were the same: sub-second response for interactive use, sub-50ms for batch scoring millions of configs nightly. We cached the rule evaluation results for configs that hadn’t changed, dramatically reducing computation for nightly runs.

What I’m Learning#

Model serving is a read-heavy, compute-heavy service that benefits from keeping models in memory, pre-computing features, and batching predictions when latency allows. The bottleneck is usually feature assembly, not model inference itself.

Have you built systems where pre-computing inputs dramatically reduced the latency of a computationally expensive operation?