object DownloadProgressTracker
Byte counts for the download currently in flight, readable from outside the JVM.
A Spark driver launched through py4j writes to a stdout that a notebook front end does not surface, so DownloadProgress's own output is invisible there however it is formatted. Python can print to the notebook, but has no view of the transfer -- it is blocked in a py4j call for its whole duration. This is the bridge: the JVM keeps the counters, and the Python side polls them and renders.
Every download on the driver shares this one set of counters. Two downloads at once (a job that builds several pipelines on threads) therefore mix their numbers in the display, which is cosmetic: each download's own data is unaffected. Accessors are plain public methods returning primitives, which is what py4j can reach.
- Alphabetic
- By Inheritance
- DownloadProgressTracker
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
bytesDone: Long
Bytes transferred so far.
Bytes transferred so far. Polled from Python.
-
def
bytesTotal: Long
Size of the object, or 0 when it is not known.
Size of the object, or 0 when it is not known. Polled from Python.
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
configuredStyle: String
The configured download_progress, normalised to bar, lines, off or auto.
The configured download_progress, normalised to bar, lines, off or auto. Python resolves auto against its own stdout, which it can see and this JVM cannot.
-
def
currentLabel: String
Name of the object being fetched, for display.
Name of the object being fetched, for display. Polled from Python.
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
isActive: Boolean
True while a download is in flight.
True while a download is in flight. Polled from Python.
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
setRenderedByCaller(value: Boolean): Unit
Set by Python around a download it draws itself, so this JVM does not draw a second copy onto the same terminal.
Set by Python around a download it draws itself, so this JVM does not draw a second copy onto the same terminal. pyspark starts the JVM with a piped stdin, so the JVM never has a console and would otherwise print one line per percent even in a terminal. Counted rather than a flag: two notebook cells downloading at once each set it, and the JVM stays quiet until both have cleared it.
-
def
snapshot: String
The whole state in one call, as
active|done|total|label.The whole state in one call, as
active|done|total|label.Polling the four accessors separately would be four py4j round trips several times a second, on a gateway the caller's own blocked download is already using.
-
def
snapshotWithGeneration: String
snapshot behind a number that every new download increments, as
generation|active|done|total|label.snapshot behind a number that every new download increments, as
generation|active|done|total|label. Python notes it when a download call starts, so it can tell that call's download from counters an earlier one left behind (even a download that starts and finishes between two polls) and can follow a second download in one call. A separate method rather than a widersnapshot, so older Python keeps parsing that. -
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()