Description
We have trained this Doc2Vec model by using Gigaword 5th Edition and English Wikipedia Dump of February 2017 over the window size of 5 and 300 dimensions. We used the Doc2VecApproach annotator that uses the Spark ML Word2Vec behind the scene to train a Word2Vec model.
Predicted Entities
How to use
document = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
token = Tokenizer()\
.setInputCols("document")\
.setOutputCol("token")
norm = Normalizer()\
.setInputCols(["token"])\
.setOutputCol("normalized")\
.setLowercase(True)
stops = StopWordsCleaner.pretrained()\
.setInputCols("normalized")\
.setOutputCol("cleanedToken")
doc2Vec = Doc2VecModel.pretrained("doc2vec_gigaword_wiki_300", "en")\
.setInputCols("cleanedToken")\
.setOutputCol("sentence_embeddings")
val document = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val tokenizer = new Tokenizer()
.setInputCols(Array("document"))
.setOutputCol("token")
val norm = new Normalizer()
.setInputCols(Array("token"))
.setOutputCol("normalized")
.setLowercase(true)
val stops = StopWordsCleaner.pretrained()
.setInputCols("normalized")
.setOutputCol("cleanedToken")
val doc2Vec = Doc2VecModel.pretrained("doc2vec_gigaword_wiki_300", "en")
.setInputCols("cleanedToken")
.setOutputCol("sentence_embeddings")
import nlu
nlu.load("en.embed_sentence.doc2vec.gigaword_wiki_300").predict("""Put your text here.""")
Model Information
| Model Name: | doc2vec_gigaword_wiki_300 |
| Compatibility: | Spark NLP 3.3.3+ |
| License: | Open Source |
| Edition: | Official |
| Input Labels: | [cleanedToken] |
| Output Labels: | [sentence_embeddings] |
| Language: | en |
Data Source
https://catalog.ldc.upenn.edu/LDC2011T07 https://dumps.wikimedia.org/enwiki/