> For the complete documentation index, see [llms.txt](https://george-jen.gitbook.io/data-science-and-apache-spark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://george-jen.gitbook.io/data-science-and-apache-spark/countvectorizer.md).

# CountVectorizer

CountVectorizer converts text documents to vectors of term counts. IDF: IDF is an Estimator which is fit on a dataset and produces an IDFModel. The IDFModel takes feature vectors (generally created from HashingTF or CountVectorizer) and scales each feature. Intuitively, it down-weights features which appear frequently in a corpus.

```
import org.apache.spark.ml.feature.{RegexTokenizer, Tokenizer}


val tokenizer = new Tokenizer().setInputCol("message")
  .setOutputCol("words")


val wordsData = tokenizer.transform(df_select)
wordsData.show(3, false)



import org.apache.spark.ml.feature.{CountVectorizer}


val count = new CountVectorizer().setInputCol("words")
   .setOutputCol("rawFeatures")


val model = count.fit(wordsData)


val featurizedData = model.transform(wordsData)


featurizedData.show(3,false)
```