問題文
The same team then wants to lay out a large table by a high-cardinality user identifier so that later joins on that identifier avoid a shuffle. Which mechanism does Apache Spark provide, and what is its restriction?
選択肢
- Sorting the table by the identifier before writing, which lets Spark skip the exchange on later joins, and the bucket count does not matter.
- Caching the table in memory, which removes the need for an exchange on later joins.
- Bucketing with a fixed number of buckets, which is applicable only to persistent tables.
- Partitioning by that identifier, which works for any destination and creates one folder per distinct value so that the join can read only the matching folders on both sides without moving any data between the executors.