TFIDF

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. final def asInstanceOf[T0]: T0

   Definition Classes

   Any

7. def calculate(tf: Int, df: Int, length: Int, numDocs: Int): Double

   Calculate TF-IDF weight.

   Calculate TF-IDF weight.

   Lucene 4.6's DefaultSimilarity TF-IDF calculation uses the formula:

   sqrt(termFreq) * (log(numDocs / (docFreq + 1)) + 1.0)

   Note: this is consistent with the MapReduce seq2sparse implementation of TF-IDF weights and is slightly different from Spark MLlib's TD-IDF calculation which is implemented as:

   termFreq * log((numDocs + 1.0) / (docFreq + 1.0))

   tf

   term freq

   df

   doc freq

   length

   Length of the document - UNUSED

   numDocs

   the total number of docs

   returns

   The TF-IDF weight as calculated by Lucene 4.6's DefaultSimilarity

   Definition Classes

   TFIDF → TermWeight

8. def clone(): AnyRef

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

9. final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

10. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

11. def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

12. final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

13. def hashCode(): Int

    Definition Classes

    AnyRef → Any

14. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

15. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

16. final def notify(): Unit

    Definition Classes

    AnyRef

17. final def notifyAll(): Unit

    Definition Classes

    AnyRef

18. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

19. def toString(): String

    Definition Classes

    AnyRef → Any

20. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

21. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

22. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )