o

com.johnsnowlabs.client.aws

DownloadProgressTracker

object DownloadProgressTracker

Byte counts for the download currently in flight, readable from outside the JVM.

A Spark driver launched through py4j writes to a stdout that a notebook front end does not surface, so DownloadProgress's own output is invisible there however it is formatted. Python can print to the notebook, but has no view of the transfer -- it is blocked in a py4j call for its whole duration. This is the bridge: the JVM keeps the counters, and the Python side polls them and renders.

Every download on the driver shares this one set of counters. Two downloads at once (a job that builds several pipelines on threads) therefore mix their numbers in the display, which is cosmetic: each download's own data is unaffected. Accessors are plain public methods returning primitives, which is what py4j can reach.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DownloadProgressTracker
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def bytesDone: Long

    Bytes transferred so far.

    Bytes transferred so far. Polled from Python.

  6. def bytesTotal: Long

    Size of the object, or 0 when it is not known.

    Size of the object, or 0 when it is not known. Polled from Python.

  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. def configuredStyle: String

    The configured download_progress, normalised to bar, lines, off or auto.

    The configured download_progress, normalised to bar, lines, off or auto. Python resolves auto against its own stdout, which it can see and this JVM cannot.

  9. def currentLabel: String

    Name of the object being fetched, for display.

    Name of the object being fetched, for display. Polled from Python.

  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def isActive: Boolean

    True while a download is in flight.

    True while a download is in flight. Polled from Python.

  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. def setRenderedByCaller(value: Boolean): Unit

    Set by Python around a download it draws itself, so this JVM does not draw a second copy onto the same terminal.

    Set by Python around a download it draws itself, so this JVM does not draw a second copy onto the same terminal. pyspark starts the JVM with a piped stdin, so the JVM never has a console and would otherwise print one line per percent even in a terminal. Counted rather than a flag: two notebook cells downloading at once each set it, and the JVM stays quiet until both have cleared it.

  21. def snapshot: String

    The whole state in one call, as active|done|total|label.

    The whole state in one call, as active|done|total|label.

    Polling the four accessors separately would be four py4j round trips several times a second, on a gateway the caller's own blocked download is already using.

  22. def snapshotWithGeneration: String

    snapshot behind a number that every new download increments, as generation|active|done|total|label.

    snapshot behind a number that every new download increments, as generation|active|done|total|label. Python notes it when a download call starts, so it can tell that call's download from counters an earlier one left behind (even a download that starts and finishes between two polls) and can follow a second download in one call. A separate method rather than a wider snapshot, so older Python keeps parsing that.

  23. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    AnyRef → Any
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped