Bucketing, Sorting and Partitioning
For file-based data source, it is also possible to bucket and sort or partition the output. Bucketing and sorting are applicable only to persistent tables:
1
peopleDF.write.bucketBy(42, "name").sortBy("age").saveAsTable("people_bucketed")
Copied!
while partitioning can be used with both save and saveAsTable when using the Dataset APIs.
1
usersDF.write.partitionBy("favorite_color").format("parquet").save("file:///tmp/namesPartByColor.parquet")
Copied!
It is possible to use both partitioning and bucketing for a single table:
1
usersDF
2
.write
3
.partitionBy("favorite_color")
4
.bucketBy(42, "name")
5
.saveAsTable("users_partitioned_bucketed")
Copied!
partitionBy creates a directory structure as described in the Partition Discovery section. Thus, it has limited applicability to columns with high cardinality.
bucketBy distributes data across a fixed number of buckets and can be used when a number of unique values is unbounded.
Last modified 1yr ago
Copy link