Packages

object ResourceDownloader

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ResourceDownloader
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. def cacheFolder: String
  2. def clearCache(request: ResourceRequest): Unit
  3. def clearCache(name: String, language: Option[String] = None, folder: String = publicLoc): Unit
  4. var communityDownloader: ResourceDownloader
  5. def downloadModel[TModel <: PipelineStage](reader: DefaultParamsReadable[TModel], request: ResourceRequest): TModel
  6. def downloadModel[TModel <: PipelineStage](reader: DefaultParamsReadable[TModel], name: String, language: Option[String] = None, folder: String = publicLoc): TModel
  7. def downloadModelDirectly(model: String, folder: String = publicLoc, unzip: Boolean = true): Unit

    Downloads a model from the default S3 bucket to the cache pretrained folder.

    Downloads a model from the default S3 bucket to the cache pretrained folder.

    model

    the name of the key in the S3 bucket or s3 URI

    folder

    the folder of the model

    unzip

    used to unzip the model, by default true

  8. def downloadPipeline(request: ResourceRequest): PipelineModel
  9. def downloadPipeline(name: String, language: Option[String] = None, folder: String = publicLoc): PipelineModel
  10. def downloadResource(request: ResourceRequest): String

    Loads resource to path

    Loads resource to path

    request

    Request for resource

    returns

    path of downloaded resource

  11. def downloadResource(name: String, language: Option[String] = None, folder: String = publicLoc): String

    Loads resource to path

    Loads resource to path

    name

    Name of Resource

    language

    Desired language of Resource

    folder

    Subfolder in s3 where to search model (e.g. medicine)

    returns

    path of downloaded resource

  12. val fileSystem: FileSystem
  13. def getDownloadSize(resourceRequest: ResourceRequest): String
  14. def getResourceDownloader(folder: String): ResourceDownloader
  15. lazy val libVersion: Version
  16. def listAvailableAnnotators(folder: String = publicLoc): List[String]
  17. def listPretrainedResources(folder: String, resourceType: ResourceType, lang: String, version: Version): List[String]
  18. def listPretrainedResources(folder: String, resourceType: ResourceType, version: Version): List[String]
  19. def listPretrainedResources(folder: String, resourceType: ResourceType, lang: String): List[String]
  20. def listPretrainedResources(folder: String, resourceType: ResourceType, annotator: Option[String] = None, lang: Option[String] = None, version: Option[Version] = None): List[String]

    Lists pretrained resource from metadata.json, depending on the set filters.

    Lists pretrained resource from metadata.json, depending on the set filters. The folder in the S3 location and the resourceType is necessary. The other filters are optional and will be ignored if not set.

    folder

    Folder in the S3 location

    resourceType

    Type of the Resource. Can Either ResourceType.MODEL, ResourceType.PIPELINE or ResourceType.NOT_DEFINED

    annotator

    Name of the model class

    lang

    Language of the model

    version

    Version that the model should be compatible with

    returns

    A list of the available resources

  21. def listPublicModels(): List[String]

    List all pretrained models in public name_lang

  22. def listPublicPipelines(): List[String]

    List all pretrained pipelines in public

  23. def listUnCategorizedResources(): List[String]

    Returns models or pipelines in metadata json which has not been categorized yet.

    Returns models or pipelines in metadata json which has not been categorized yet.

    returns

    list of models or pipelines which are not categorized in metadata json

  24. var privateDownloader: ResourceDownloader
  25. var publicDownloader: ResourceDownloader
  26. val publicLoc: String
  27. def publicResourceString(annotator: Option[String] = None, lang: Option[String] = None, version: Option[String] = Some(Build.version), resourceType: ResourceType): String
  28. def resetResourceDownloader(): Unit

    Reset the cache and recreate ResourceDownloader S3 credentials

  29. def s3Bucket: String
  30. def s3BucketCommunity: String
  31. def s3Path: String
  32. def showAvailableAnnotators(folder: String = publicLoc): Unit
  33. def showPublicModels(annotator: String, lang: String, version: String): Unit

    Prints all pretrained models for a particular annotator, that are compatible with a version of Spark NLP.

    Prints all pretrained models for a particular annotator, that are compatible with a version of Spark NLP.

    annotator

    Name of the model class, for example "NerDLModel"

    lang

    Language of the pretrained models to display, for example "en"

    version

    Version of Spark NLP that the model should be compatible with, for example "3.2.3"

  34. def showPublicModels(annotator: String, lang: String): Unit

    Prints all pretrained models for a particular annotator model, that are compatible with this version of Spark NLP.

    Prints all pretrained models for a particular annotator model, that are compatible with this version of Spark NLP.

    annotator

    Name of the annotator class

    lang

    Language of the pretrained models to display

  35. def showPublicModels(annotator: String): Unit

    Prints all pretrained models for a particular annotator model, that are compatible with this version of Spark NLP.

    Prints all pretrained models for a particular annotator model, that are compatible with this version of Spark NLP.

    annotator

    Name of the annotator class

  36. def showPublicModels(annotator: Option[String] = None, lang: Option[String] = None, version: Option[String] = Some(Build.version)): Unit

    Prints all pretrained models for a particular annotator model, that are compatible with a version of Spark NLP.

    Prints all pretrained models for a particular annotator model, that are compatible with a version of Spark NLP. If any of the optional arguments are not set, the filter is not considered.

    annotator

    Name of the model class, for example "NerDLModel"

    lang

    Language of the pretrained models to display, for example "en"

    version

    Version of Spark NLP that the model should be compatible with, for example "3.2.3"

  37. def showPublicPipelines(lang: String, version: String): Unit

    Prints all Pipelines available for a language and a version of Spark NLP.

    Prints all Pipelines available for a language and a version of Spark NLP.

    lang

    Language of the Pipeline

    version

    Version of Spark NLP

  38. def showPublicPipelines(lang: String): Unit

    Prints all Pipelines available for a language and this version of Spark NLP.

    Prints all Pipelines available for a language and this version of Spark NLP.

    lang

    Language of the Pipeline

  39. def showPublicPipelines(lang: Option[String] = None, version: Option[String] = Some(Build.version)): Unit

    Prints all Pipelines available for a language and a version of Spark NLP.

    Prints all Pipelines available for a language and a version of Spark NLP. By default shows all languages and uses the current version of Spark NLP.

    lang

    Language of the Pipeline

    version

    Version of Spark NLP

  40. def showString(list: List[String], resourceType: ResourceType): String
  41. def showUnCategorizedResources(lang: String, version: String): Unit
  42. def showUnCategorizedResources(lang: String): Unit
  43. lazy val sparkVersion: Version