Generic Load/Save Functions
The default data source (parquet unless otherwise configured by spark.sql.sources.default) will be used for all operations.
val usersDF = spark.read.load("file:///home/dv6/spark/spark/examples/src/main/resources/users.parquet")
usersDF.select("name", "favorite_color").write.save("file:///tmp/namesAndFavColors.parquet")
or you can write in one line
spark.read.load("file:///home/dv6/spark/spark/examples/src/main/resources/users.parquet")
.select("name", "favorite_color")
.write.save("file:///tmp/namesAndFavColors.parquet")
Last modified 3yr ago