Interaction

​
Interaction is a Transformer which takes vector or double-valued columns, and generates a single vector column that contains the product of all combinations of one value from each input column.
For example, if you have 2 vector type columns each of which has 3 dimensions as input columns, then you’ll get a 9-dimensional vector as the output column.
It is a Cartesian product between 2 vectors
Examples
Assume that we have the following DataFrame with the columns “id1”, “vec1”, and “vec2”:
id1
vec1
vec2
1
[1.0,2.0,3.0]
[8.0,4.0,5.0]
2
[4.0,3.0,8.0]
[7.0,9.0,8.0]
3
[6.0,1.0,9.0]
[2.0,3.0,6.0]
4
[10.0,8.0,6.0]
[9.0,4.0,5.0]
5
[9.0,2.0,7.0]
[10.0,7.0,3.0]
6
[1.0,1.0,4.0]
[2.0,8.0,4.0]
Applying Interaction with those input columns, then interactedCol as the output column contains:
id1
vec1
vec2
interactedCol
1
[1.0,2.0,3.0]
[8.0,4.0,5.0]
[8.0,4.0,5.0,16.0,8.0,10.0,24.0,12.0,15.0]
2
[4.0,3.0,8.0]
[7.0,9.0,8.0]
[56.0,72.0,64.0,42.0,54.0,48.0,112.0,144.0,128.0]
3
[6.0,1.0,9.0]
[2.0,3.0,6.0]
[36.0,54.0,108.0,6.0,9.0,18.0,54.0,81.0,162.0]
4
[10.0,8.0,6.0]
[9.0,4.0,5.0]
[360.0,160.0,200.0,288.0,128.0,160.0,216.0,96.0,120.0]
5
[9.0,2.0,7.0]
[10.0,7.0,3.0]
[450.0,315.0,135.0,100.0,70.0,30.0,350.0,245.0,105.0]
6
[1.0,1.0,4.0]
[2.0,8.0,4.0]
[12.0,48.0,24.0,12.0,48.0,24.0,48.0,192.0,96.0]
import org.apache.spark.ml.feature.Interaction
import org.apache.spark.ml.feature.VectorAssembler
​
val df = spark.createDataFrame(Seq(
  (1, 1, 2, 3, 8, 4, 5),
  (2, 4, 3, 8, 7, 9, 8),
  (3, 6, 1, 9, 2, 3, 6),
  (4, 10, 8, 6, 9, 4, 5),
  (5, 9, 2, 7, 10, 7, 3),
  (6, 1, 1, 4, 2, 8, 4)
)).toDF("id1", "id2", "id3", "id4", "id5", "id6", "id7")
​
val assembler1 = new VectorAssembler().
  setInputCols(Array("id2", "id3", "id4")).
  setOutputCol("vec1")
​
val assembled1 = assembler1.transform(df)
​
val assembler2 = new VectorAssembler().
  setInputCols(Array("id5", "id6", "id7")).
  setOutputCol("vec2")
​
val assembled2 = assembler2.transform(assembled1).select("id1", "vec1", "vec2")
​
val interaction = new Interaction()
  .setInputCols(Array("id1", "vec1", "vec2"))
  .setOutputCol("interactedCol")
​
val interacted = interaction.transform(assembled2)
​
interacted.show(truncate = false)
​
/*
Output:
+---+--------------+--------------+------------------------------------------------------+
|id1|vec1          |vec2          |interactedCol                                         |
+---+--------------+--------------+------------------------------------------------------+
|1  |[1.0,2.0,3.0] |[8.0,4.0,5.0] |[8.0,4.0,5.0,16.0,8.0,10.0,24.0,12.0,15.0]            |
|2  |[4.0,3.0,8.0] |[7.0,9.0,8.0] |[56.0,72.0,64.0,42.0,54.0,48.0,112.0,144.0,128.0]     |
|3  |[6.0,1.0,9.0] |[2.0,3.0,6.0] |[36.0,54.0,108.0,6.0,9.0,18.0,54.0,81.0,162.0]        |
|4  |[10.0,8.0,6.0]|[9.0,4.0,5.0] |[360.0,160.0,200.0,288.0,128.0,160.0,216.0,96.0,120.0]|
|5  |[9.0,2.0,7.0] |[10.0,7.0,3.0]|[450.0,315.0,135.0,100.0,70.0,30.0,350.0,245.0,105.0] |
|6  |[1.0,1.0,4.0] |[2.0,8.0,4.0] |[12.0,48.0,24.0,12.0,48.0,24.0,48.0,192.0,96.0]       |
+---+--------------+--------------+------------------------------------------------------+
​
​
​
*/

VectorIndexer

Normalizer

Last modified 3yr ago

id1	vec1	vec2
1	[1.0,2.0,3.0]	[8.0,4.0,5.0]
2	[4.0,3.0,8.0]	[7.0,9.0,8.0]
3	[6.0,1.0,9.0]	[2.0,3.0,6.0]
4	[10.0,8.0,6.0]	[9.0,4.0,5.0]
5	[9.0,2.0,7.0]	[10.0,7.0,3.0]
6	[1.0,1.0,4.0]	[2.0,8.0,4.0]

id1	vec1	vec2	interactedCol
1	[1.0,2.0,3.0]	[8.0,4.0,5.0]	[8.0,4.0,5.0,16.0,8.0,10.0,24.0,12.0,15.0]
2	[4.0,3.0,8.0]	[7.0,9.0,8.0]	[56.0,72.0,64.0,42.0,54.0,48.0,112.0,144.0,128.0]
3	[6.0,1.0,9.0]	[2.0,3.0,6.0]	[36.0,54.0,108.0,6.0,9.0,18.0,54.0,81.0,162.0]
4	[10.0,8.0,6.0]	[9.0,4.0,5.0]	[360.0,160.0,200.0,288.0,128.0,160.0,216.0,96.0,120.0]
5	[9.0,2.0,7.0]	[10.0,7.0,3.0]	[450.0,315.0,135.0,100.0,70.0,30.0,350.0,245.0,105.0]
6	[1.0,1.0,4.0]	[2.0,8.0,4.0]	[12.0,48.0,24.0,12.0,48.0,24.0,48.0,192.0,96.0]