org.apache.mahout.sparkbindings.drm

CheckpointedDrmSpark

class CheckpointedDrmSpark[K] extends CheckpointedDrm[K]

Spark-specific optimizer-checkpointed DRM.

K

matrix key type (e.g. the keys of sequence files once persisted)

Linear Supertypes
CheckpointedDrm[K], DrmLike[K], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. CheckpointedDrmSpark
  2. CheckpointedDrm
  3. DrmLike
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CheckpointedDrmSpark(rddInput: DrmRddInput[K], _nrow: Long = -1L, _ncol: Int = -1, cacheHint: CacheHint = ..., partitioningTag: Long = scala.util.Random.nextLong(), _canHaveMissingRows: Boolean = false)(implicit arg0: ClassTag[K])

    rddInput

    underlying rdd to wrap over.

    _nrow

    number of rows; if unspecified, we will compute with an inexpensive traversal.

    _ncol

    number of columns; if unspecified, we will try to guess with an inexpensive traversal.

    cacheHint

    cache level to use. (Implementors usually want to override the default!)

    partitioningTag

    unique partitioning tag. Used to detect identically partitioned operands.

    _canHaveMissingRows

    true if the matrix is int-keyed, and if it also may have missing rows (will require a lazy fix for some physical operations.

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def cache(): CheckpointedDrmSpark[K]

  8. val cacheHint: CacheHint

    cache level to use.

    cache level to use. (Implementors usually want to override the default!)

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  9. lazy val canHaveMissingRows: Boolean

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  10. def checkpoint(cacheHint: CacheHint): CheckpointedDrm[K]

    Action operator -- does not necessary means Spark action; but does mean running BLAS optimizer and writing down Spark graph lineage since last checkpointed DRM.

    Action operator -- does not necessary means Spark action; but does mean running BLAS optimizer and writing down Spark graph lineage since last checkpointed DRM.

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  11. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  12. def collect: Matrix

    Collecting DRM to fron-end in-core Matrix.

    Collecting DRM to fron-end in-core Matrix.

    If key in DRM is Int, then matrix is collected using key as row index. Otherwise, order of rows in result is undefined but key.toString is applied as rowLabelBindings of the in-core matrix .

    Note that this pre-allocates target matrix and then assigns collected RDD to it thus this likely would require about 2 times the RDD memory

    returns

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  13. def computeNCol: Int

    Attributes
    protected
  14. def computeNNonZero: Long

    Attributes
    protected
  15. def computeNRow: Long

    Attributes
    protected
  16. val context: DistributedContext

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  17. def dfsWrite(path: String): Unit

    Dump matrix as computed Mahout's DRM into specified (HD)FS path

    Dump matrix as computed Mahout's DRM into specified (HD)FS path

    path

    output path to dump Matrix to

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  18. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  19. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  20. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  21. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  22. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  23. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  24. def keyClassTag: ClassTag[K]

    Explicit extraction of key class Tag

    Explicit extraction of key class Tag

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  25. lazy val ncol: Int

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  26. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  27. def newRowCardinality(n: Int): CheckpointedDrm[K]

    Changes the number of rows in the DRM without actually touching the underlying data.

    Changes the number of rows in the DRM without actually touching the underlying data. Used to redimension a DRM after it has been created, which implies some blank, non-existent rows.

    n

    new row dimension

    returns

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  28. final def notify(): Unit

    Definition Classes
    AnyRef
  29. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  30. lazy val nrow: Long

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  31. val partitioningTag: Long

    unique partitioning tag.

    unique partitioning tag. Used to detect identically partitioned operands.

    Attributes
    protected[org.apache.mahout]
    Definition Classes
    CheckpointedDrmSpark → DrmLike
  32. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  33. def toString(): String

    Definition Classes
    AnyRef → Any
  34. def uncache(): CheckpointedDrmSpark.this.type

    if matrix was previously persisted into cache, delete cached representation

    if matrix was previously persisted into cache, delete cached representation

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  35. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from CheckpointedDrm[K]

Inherited from DrmLike[K]

Inherited from AnyRef

Inherited from Any

Ungrouped