PrefixSpan
PrefixSpan is a sequential pattern mining algorithm described in Pei et al., Mining Sequential Patterns by Pattern-Growth: The PrefixSpan Approach. We refer the reader to the referenced paper for formalizing the sequential pattern mining problem.
spark.ml's PrefixSpan implementation takes the following parameters:
minSupport: the minimum support required to be considered a frequent sequential pattern.
maxPatternLength: the maximum length of a frequent sequential pattern. Any frequent pattern exceeding this length will not be included in the results.
maxLocalProjDBSize: the maximum number of items allowed in a prefix-projected database before local iterative processing of the projected database begins. This parameter should be tuned with respect to the size of your executors.
sequenceCol: the name of the sequence column in dataset (default "sequence"), rows with nulls in this column are ignored.
import org.apache.spark.ml.fpm.PrefixSpan
val smallTestData = Seq(
Seq(Seq(1, 2), Seq(3)),
Seq(Seq(1), Seq(3, 2), Seq(1, 2)),
Seq(Seq(1, 2), Seq(5)),
Seq(Seq(6)))
val df = smallTestData.toDF("sequence")
val result = new PrefixSpan()
.setMinSupport(0.5)
.setMaxPatternLength(5)
.setMaxLocalProjDBSize(32000000)
.findFrequentSequentialPatterns(df)
.show()
/*
Output:
+----------+----+
| sequence|freq|
+----------+----+
| [[3]]| 2|
| [[2]]| 3|
| [[1]]| 3|
| [[1, 2]]| 3|
|[[1], [3]]| 2|
+----------+----+
*/
Last modified 3yr ago