Word2Vec Gigaword and Wikipedia - 300 dimensions

Description

We have trained this Word2Vec model by using Gigaword 5th Edition and English Wikipedia Dump of February 2017 over the window size of 5 and 300 dimensions. We used the Word2VecApproach annotator that uses the Spark ML Word2Vec behind the scene to train a Word2Vec model.

Predicted Entities

Download Copy S3 URI

How to use

document = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")

token = Tokenizer()\
.setInputCols("document")\
.setOutputCol("token")

norm = Normalizer()\
.setInputCols(["token"])\
.setOutputCol("normalized")\
.setLowercase(True)

stops = StopWordsCleaner.pretrained()\
.setInputCols("normalized")\
.setOutputCol("cleanedToken")

doc2Vec = Word2VecModel.pretrained("word2vec_gigaword_wiki_300", "en")\
.setInputCols("cleanedToken")\
.setOutputCol("sentence_embeddings")
val document = new DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val tokenizer = new Tokenizer()
.setInputCols(Array("document"))
.setOutputCol("token")

val norm = new Normalizer()
.setInputCols(Array("token"))
.setOutputCol("normalized")
.setLowercase(true)

val stops = StopWordsCleaner.pretrained()
.setInputCols("normalized")
.setOutputCol("cleanedToken")

val doc2Vec = Word2VecModel.pretrained("word2vec_gigaword_wiki_300", "en")
.setInputCols("cleanedToken")
.setOutputCol("sentence_embeddings")
import nlu
nlu.load("en.embed.word2vec.gigaword_wiki").predict("""Put your text here.""")

Model Information

Model Name: word2vec_gigaword_wiki_300
Compatibility: Spark NLP 3.4.0+
License: Open Source
Edition: Official
Input Labels: [token]
Output Labels: [embeddings]
Language: en
Size: 326.7 MB

Data Source

https://catalog.ldc.upenn.edu/LDC2011T07

https://dumps.wikimedia.org/enwiki/