Description
We have trained this Word2Vec model by using Gigaword 5th Edition over the window size of 5 and 300 dimensions. We used the Word2VecApproach annotator that uses the Spark ML Word2Vec behind the scene to train a Word2Vec model.
Predicted Entities
How to use
document = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
token = Tokenizer()\
.setInputCols("document")\
.setOutputCol("token")
norm = Normalizer()\
.setInputCols(["token"])\
.setOutputCol("normalized")\
.setLowercase(True)
stops = StopWordsCleaner.pretrained()\
.setInputCols("normalized")\
.setOutputCol("cleanedToken")
doc2Vec = Word2VecModel.pretrained("word2vec_gigaword_300", "en")\
.setInputCols("cleanedToken")\
.setOutputCol("sentence_embeddings")
val document = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val tokenizer = new Tokenizer()
.setInputCols(Array("document"))
.setOutputCol("token")
val norm = new Normalizer()
.setInputCols(Array("token"))
.setOutputCol("normalized")
.setLowercase(true)
val stops = StopWordsCleaner.pretrained()
.setInputCols("normalized")
.setOutputCol("cleanedToken")
val doc2Vec = Word2VecModel.pretrained("word2vec_gigaword_300", "en")
.setInputCols("cleanedToken")
.setOutputCol("sentence_embeddings")
import nlu
nlu.load("en.embed.word2vec.gigaword").predict("""Put your text here.""")
Model Information
| Model Name: | word2vec_gigaword_300 |
| Compatibility: | Spark NLP 3.4.0+ |
| License: | Open Source |
| Edition: | Official |
| Input Labels: | [token] |
| Output Labels: | [embeddings] |
| Language: | en |
| Size: | 327.5 MB |