doc2vec

Kohei Watanabe

2026-07-28

In this article, wordvector is used to train doc2vec on quanteda’s tokens objects.

Prepare data

Download the corpus of news summaries.

# Load data
f <- tempfile()
download.file('https://www.dropbox.com/s/e19kslwhuu9yc2z/yahoo-news.RDS?dl=1', 
              f, mode = "wb")

library(quanteda)
## Package version: 4.4
## Unicode version: 15.1
## ICU version: 74.2
## Parallel computing: disabled
## See https://quanteda.io for tutorials and examples.
library(wordvector)
quanteda_options(verbose = TRUE)

# Construct corpus
dat <- readRDS(f)
dat$text <- paste0(dat$head, ". ", dat$body)
corp <- corpus(dat, text_field = 'text')

# Tokenize
toks <- tokens(corp, remove_punct = TRUE, remove_symbols = TRUE) %>% 
    tokens_remove(stopwords("en", "marimo"), padding = TRUE) %>% 
    tokens_select("^[a-zA-Z-]+$", valuetype = "regex", case_insensitive = FALSE,
                  padding = TRUE)
## Creating a tokens from a corpus object...
##  ...starting tokenization
##  ...tokenizing 1 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 2 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 3 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 4 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 5 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 6 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 7 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...removing separators, punctuation, symbols
##  ...298,565 unique types
##  ...complete, elapsed time: 86.4 seconds.
## Finished constructing tokens from 656,334 documents
## tokens_remove() changed from 45,194,192 tokens (656,334 documents) to 28,232,774 tokens (656,334 documents)
## tokens_keep() changed from 28,232,774 tokens (656,334 documents) to 26,564,509 tokens (656,334 documents)

Train doc2vec

textmodel_doc2vec() supports both dm (distributed memory) and dbow (distributed bag-of-words) models.

# Set the number of processors
options(wordvector_threads = 16)

# Train doc2vec
dov <- textmodel_doc2vec(toks, dim = 50, type = "dm", min_count = 5, verbose = TRUE)
## Training distributed memory model with 50 dimensions
##  ...using 16 threads for distributed computing
##  ...initializing
##  ...negative sampling in 10 iterations
##  ......iteration 1 elapsed time: 18.37 seconds (alpha: 0.0456)
##  ......iteration 2 elapsed time: 38.51 seconds (alpha: 0.0407)
##  ......iteration 3 elapsed time: 59.38 seconds (alpha: 0.0356)
##  ......iteration 4 elapsed time: 77.16 seconds (alpha: 0.0313)
##  ......iteration 5 elapsed time: 95.55 seconds (alpha: 0.0269)
##  ......iteration 6 elapsed time: 116.96 seconds (alpha: 0.0218)
##  ......iteration 7 elapsed time: 138.85 seconds (alpha: 0.0167)
##  ......iteration 8 elapsed time: 160.27 seconds (alpha: 0.0116)
##  ......iteration 9 elapsed time: 180.52 seconds (alpha: 0.0066)
##  ......iteration 10 elapsed time: 191.99 seconds (alpha: 0.0038)
##  ...complete

Since the distributed memory model has hidden layers for documents and words, you can extract document and word vectors using as.matrix()

# Extract document vector
dim(as.matrix(dov, layer = "documents"))
## [1] 656334     50

# Extract word vector
dim(as.matrix(dov, layer = "words"))
## [1] 79259    50

Predict probability

If probabitliy() is applied to a fitted doc2vec model, you receive the predicted probability of the words in each document.

head(probability(dov, c("bad", "good"), mode = "numeric", layer = "documents"))
##             bad       good
## text1 0.2356755 0.33294834
## text2 0.2033403 0.17415359
## text3 0.1424676 0.07495828
## text4 0.4886010 0.79879816
## text5 0.4922210 0.45500447
## text6 0.7270843 0.62980089