doc2vec

Kohei Watanabe

2026-09-10

In this article, wordvector is used to train doc2vec on quanteda’s tokens objects.

Prepare data

Download the corpus of news summaries.

# Load data
f <- tempfile()
download.file('https://www.dropbox.com/s/e19kslwhuu9yc2z/yahoo-news.RDS?dl=1', 
              f, mode = "wb")

library(quanteda)
## Package version: 4.5.0
## Unicode version: 15.1
## ICU version: 74.2
## Parallel computing: disabled
## See https://quanteda.io for tutorials and examples.
library(wordvector)
quanteda_options(verbose = TRUE)

# Construct corpus
dat <- readRDS(f)
dat$text <- paste0(dat$head, ". ", dat$body)
corp <- corpus(dat, text_field = 'text')

# Tokenize
toks <- tokens(corp, remove_punct = TRUE, remove_symbols = TRUE) %>% 
    tokens_remove(stopwords("en", "marimo"), padding = TRUE) %>% 
    tokens_select("^[a-zA-Z-]+$", valuetype = "regex", case_insensitive = FALSE,
                  padding = TRUE)
## Creating a tokens from a corpus object...
##  ...starting tokenization
##  ...tokenizing 1 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 2 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 3 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 4 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 5 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 6 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...tokenizing 7 of 7 blocks
##  ...preserving hyphens
##  ...preserving elisions
##  ...preserving social media tags (#, @)
##  ...removing separators, punctuation, symbols
##  ...298,565 unique types
##  ...complete, elapsed time: 94.2 seconds.
## Finished constructing tokens from 656,334 documents
## tokens_remove() changed from 298,565 types (656,334 documents, 45,194,192 tokens) to 298,002 types (656,334 documents, 28,232,774 tokens)
## tokens_keep() changed from 298,002 types (656,334 documents, 28,232,774 tokens) to 239,782 types (656,334 documents, 26,564,509 tokens)

Train doc2vec

textmodel_doc2vec() supports both dm (distributed memory) and dbow (distributed bag-of-words) models.

# Set the number of processors
options(wordvector_threads = 16)

# Train doc2vec
dov <- textmodel_doc2vec(toks, dim = 50, type = "dm", min_count = 5, verbose = TRUE)
## Training distributed memory model with 50 dimensions
##  ...using 16 threads for distributed computing
##  ...initializing
##  ...negative sampling in 10 iterations
##  ......iteration 1 elapsed time: 18.74 seconds (alpha: 0.0451)
##  ......iteration 2 elapsed time: 35.65 seconds (alpha: 0.0408)
##  ......iteration 3 elapsed time: 54.21 seconds (alpha: 0.0361)
##  ......iteration 4 elapsed time: 70.98 seconds (alpha: 0.0319)
##  ......iteration 5 elapsed time: 89.72 seconds (alpha: 0.0269)
##  ......iteration 6 elapsed time: 105.75 seconds (alpha: 0.0226)
##  ......iteration 7 elapsed time: 125.52 seconds (alpha: 0.0174)
##  ......iteration 8 elapsed time: 145.01 seconds (alpha: 0.0122)
##  ......iteration 9 elapsed time: 162.84 seconds (alpha: 0.0075)
##  ......iteration 10 elapsed time: 180.08 seconds (alpha: 0.0028)
##  ...complete

Since the distributed memory model has hidden layers for documents and words, you can extract document and word vectors using as.matrix()

# Extract document vector
dim(as.matrix(dov, layer = "documents"))
## [1] 656334     50

# Extract word vector
dim(as.matrix(dov, layer = "words"))
## [1] 79259    50

Predict probability

If probabitliy() is applied to a fitted doc2vec model, you receive the predicted probability of the words in each document.

head(probability(dov, c("bad", "good"), mode = "numeric", layer = "documents"))
##             bad       good
## text1 0.1722140 0.28104755
## text2 0.2655658 0.23984263
## text3 0.1134697 0.04838086
## text4 0.4808392 0.77852945
## text5 0.3289697 0.31371197
## text6 0.6684403 0.63600217