Generic Load/Save Functions
The default data source (parquet unless otherwise configured by spark.sql.sources.default) will be used for all operations.
1
val usersDF = spark.read.load("file:///home/dv6/spark/spark/examples/src/main/resources/users.parquet")
2
usersDF.select("name", "favorite_color").write.save("file:///tmp/namesAndFavColors.parquet")
Copied!
or you can write in one line
1
spark.read.load("file:///home/dv6/spark/spark/examples/src/main/resources/users.parquet")
2
.select("name", "favorite_color")
3
.write.save("file:///tmp/namesAndFavColors.parquet")
Copied!
Last modified 1yr ago
Copy link