問題文
A team is choosing the partition size for a distributed job on devices with 48 GB of memory each. The job does large-scale sorting and joining. Which target does the documented guidance give?
選択肢
- The full capacity of a single device, so that each task processes the largest possible amount of data in one pass and the total number of tasks is minimized, which is the setting that gives the highest utilization for each individual task in the graph.
- Between one half and one quarter of a device's capacity, since anything smaller produces so many tasks that the scheduler becomes the bottleneck, and a partition that fills much of the device keeps each shuffle inside one allocation.
- A fixed 64 MB regardless of the device.
- Between one thirty-second and one sixteenth of a single device's capacity, because shuffle-intensive work needs headroom for the data that is in flight during the all-to-all step.