問題文
A service must keep the 95th percentile latency under 20 milliseconds and also raise throughput. Which combination should be tried, and how should it be validated?
選択肢
- Enable request batching only and leave the instance count at one, because batching is what raises throughput and adding instances beyond one merely divides the same device between more copies of the model, which increases the latency of every request without adding any capacity at all.
- Increase the maximum batch size as far as possible, since larger batches always improve throughput, and the latency percentile can be brought back under the limit afterwards by adding hardware to the pool.
- Reduce the concurrency to one, which minimizes latency.
- Enable request batching and increase the number of model instances together, then sweep the request concurrency and check both throughput and the latency percentile at each point.