VectorIndexer
VectorIndexer helps index categorical features in datasets of Vectors. It can both automatically decide which features are categorical and convert original values to category indices. Specifically, it does the following:
Take an input column of type Vector and a parameter maxCategories. Decide which features should be categorical based on the number of distinct values, where features with at most maxCategories are declared categorical. Compute 0-based category indices for each categorical feature. Index categorical features and transform original feature values to indices. Indexing categorical features allows algorithms such as Decision Trees and Tree Ensembles to treat categorical features appropriately, improving performance.
Examples
In the example below, read in a dataset of labeled points and then use VectorIndexer to decide which features should be treated as categorical. Then transform the categorical feature values to their indices. This transformed data could then be passed to algorithms such as DecisionTreeRegressor that handle categorical features.
1
import org.apache.spark.ml.feature.VectorIndexer
2
​
3
val data = spark.read.format("libsvm").load("file:///opt/spark/data/mllib/sample_libsvm_data.txt")
4
​
5
val indexer = new VectorIndexer()
6
.setInputCol("features")
7
.setOutputCol("indexed")
8
.setMaxCategories(10)
9
​
10
val indexerModel = indexer.fit(data)
11
​
12
val categoricalFeatures: Set[Int] = indexerModel.categoryMaps.keys.toSet
13
println(s"Chose ${categoricalFeatures.size} " +
14
s"categorical features: ${categoricalFeatures.mkString(", ")}")
15
​
16
// Create new column "indexed" with categorical values transformed to indices
17
val indexedData = indexerModel.transform(data)
18
indexedData.show()
19
​
20
/*
21
Output:
22
Chose 351 categorical features: 645, 69, 365, 138, 101, 479, 333, 249, 0, 555, 666, 88, 170, 115, 276, 308, 5, 449, 120, 247, 614, 677, 202, 10, 56, 533, 142, 500, 340, 670, 174, 42, 417, 24, 37, 25, 257, 389, 52, 14, 504, 110, 587, 619, 196, 559, 638, 20, 421, 46, 93, 284, 228, 448, 57, 78, 29, 475, 164, 591, 646, 253, 106, 121, 84, 480, 147, 280, 61, 221, 396, 89, 133, 116, 1, 507, 312, 74, 307, 452, 6, 248, 60, 117, 678, 529, 85, 201, 220, 366, 534, 102, 334, 28, 38, 561, 392, 70, 424, 192, 21, 137, 165, 33, 92, 229, 252, 197, 361, 65, 97, 665, 583, 285, 224, 650, 615, 9, 53, 169, 593, 141, 610, 420, 109, 256, 225, 339, 77, 193, 669, 476, 642, 637, 590, 679, 96, 393, 647, 173, 13, 41, 503, 134, 73, 105, 2, 508, 311, 558, 674, 530, 586, 618, 166, 32, 34, 148, 45, 161, 279, 64, 689, 17, 149, 584, 562, 176, 423, 191, 22, 44, 59, 118, 281, 27, 641, 71, 391, 12, 445, 54, 313, 611, 144, 49, 335, 86, 672, 172, 113, 681, 219, 419, 81, 230, 362, 451, 76, 7, 39, 649, 98, 616, 477, 367, 535, 103, 140, 621, 91, 66, 251, 668, 198, 108, 278, 223, 394, 306, 135, 563, 226, 3, 505, 80, 167, 35, 473, 675, 589, 162, 531, 680, 255, 648, 112, 617, 194, 145, 48, 557, 690, 63, 640, 18, 282, 95, 310, 50, 67, 199, 673, 16, 585, 502, 338, 643, 31, 336, 613, 11, 72, 175, 446, 612, 143, 43, 250, 231, 450, 99, 363, 556, 87, 203, 671, 688, 104, 368, 588, 40, 304, 26, 258, 390, 55, 114, 171, 139, 418, 23, 8, 75, 119, 58, 667, 478, 536, 82, 620, 447, 36, 168, 146, 30, 51, 190, 19, 422, 564, 305, 107, 4, 136, 506, 79, 195, 474, 664, 532, 94, 283, 395, 332, 528, 644, 47, 15, 163, 200, 68, 62, 277, 691, 501, 90, 111, 254, 227, 337, 122, 83, 309, 560, 639, 676, 222, 592, 364, 100
23
+-----+--------------------+--------------------+
24
|label| features| indexed|
25
+-----+--------------------+--------------------+
26
| 0.0|(692,[127,128,129...|(692,[127,128,129...|
27
| 1.0|(692,[158,159,160...|(692,[158,159,160...|
28
| 1.0|(692,[124,125,126...|(692,[124,125,126...|
29
| 1.0|(692,[152,153,154...|(692,[152,153,154...|
30
| 1.0|(692,[151,152,153...|(692,[151,152,153...|
31
| 0.0|(692,[129,130,131...|(692,[129,130,131...|
32
| 1.0|(692,[158,159,160...|(692,[158,159,160...|
33
| 1.0|(692,[99,100,101,...|(692,[99,100,101,...|
34
| 0.0|(692,[154,155,156...|(692,[154,155,156...|
35
| 0.0|(692,[127,128,129...|(692,[127,128,129...|
36
| 1.0|(692,[154,155,156...|(692,[154,155,156...|
37
| 0.0|(692,[153,154,155...|(692,[153,154,155...|
38
| 0.0|(692,[151,152,153...|(692,[151,152,153...|
39
| 1.0|(692,[129,130,131...|(692,[129,130,131...|
40
| 0.0|(692,[154,155,156...|(692,[154,155,156...|
41
| 1.0|(692,[150,151,152...|(692,[150,151,152...|
42
| 0.0|(692,[124,125,126...|(692,[124,125,126...|
43
| 0.0|(692,[152,153,154...|(692,[152,153,154...|
44
| 1.0|(692,[97,98,99,12...|(692,[97,98,99,12...|
45
| 1.0|(692,[124,125,126...|(692,[124,125,126...|
46
+-----+--------------------+--------------------+
47
only showing top 20 rows
48
​
49
​
50
*/
Copied!
Last modified 1yr ago
Copy link