Our forecast model runs on a two-core CPU instance and answers in under 80 ms. Here is what made that possible.
Batch small, batch often
Micro-batching requests for 10 ms gave us four times the throughput with no visible latency cost.
Export once
TorchScript removed Python overhead from the hot path and halved cold-start time.
Pin threads to cores
Warm the model on boot
Cache by input hash