Skip to contents

For users who are comfortable with writing their own posterior samplers, dbarts makes it easy to incorporate a linear BART component in a Gibbs sampler. In the example below, we fit a functional mixture model, where observations may have come from one of two underlying functions and class membership is unobserved.

Model

Performing inference conditional on xx, we assume that:

Y∣Z=z∼N(f(x,z),σ2),Y\mid Z = z \sim \mathrm{N}\left(f(x, z), \sigma^2 \right),Z∼Bern(p),Z \sim \mathrm{Bern}(p),

and that ZZ is unobserved. To make the model fully Bayesian, we impose a Beta(1,1)\mathrm{Beta}(1, 1) prior on pp. ff and σ2\sigma^2 have implicit priors defined by BART.

Conditional Posteriors

To implement a Gibbs sampler, we will have to produce updates for ZZ and pp. After writing down the joint distribution of ZZ, YY, pp, ff, and σ2\sigma^2, we have:

  • P(Z=1∣Y=y,p,f,σ2)∝ϕ(y,f(x,1),σ)⋅pP(Z = 1 \mid Y = y, p, f, \sigma^2) \propto \phi(y, f(x, 1), \sigma) \cdot p
  • P(Z=0∣Y=y,p,f,σ2)∝ϕ(y,f(x,0),σ)⋅(1−p)P(Z = 0 \mid Y = y, p, f, \sigma^2) \propto \phi(y, f(x, 0), \sigma) \cdot (1 - p)
  • p∣Z=z∼Beta(1+∑zi,1+∑(1−zi))p \mid Z = z\sim \mathrm{Beta}(1 + \sum z_i, 1 + \sum (1 - z_i))

where ϕ\phi is the density of a normal distribution. This yields an overall strategy of:

  1. Draw a sample from BART
    1. Draw samples of f(x,1)f(x, 1) and f(x,0)f(x, 0)
    2. Draw a sample of σ2\sigma^2
  2. Use f(x,1)f(x, 1), f(x,0)f(x, 0), and pp to draw a sample of ZZ
  3. Use ZZ to draw a sample of pp

Simulated Data

To illustrate, we create some toy data. The simulated data is visualized later, together with the fitted model.

# Underlying functions
f0 <- function(x) 90 + exp(0.06 * x)
f1 <- function(x) 72 + 3 * sqrt(x)

set.seed(2793)

# Generate true values.
n <- 120
p.0 <- 0.5

z.0 <- rbinom(n, 1, p.0)
n1 <- sum(z.0); n0 <- n - n1

# In order to make the problem more interesting, x is confounded with both
# y and z.
x <- numeric(n)
x[z.0 == 0] <- rnorm(n0, 20, 10)
x[z.0 == 1] <- rnorm(n1, 40, 10)
y <- numeric(n)
y[z.0 == 0] <- f0(x[z.0 == 0])
y[z.0 == 1] <- f1(x[z.0 == 1])
y <- y + rnorm(n)

data_train <- data.frame(y = y, x = x, z =     z.0)
data_test  <- data.frame(       x = x, z = 1 - z.0)

Implementing a Gibbs Sampler

For this specific model, the sampler needs to be updated with new predictor variables/covariates. This can be done with a dbartsSampler reference class object by calling the sampler$setPredictor function. As, given any Z=zZ = z, the sampler also needs to evaluate f(x,1−z)f(x, 1 - z), we utilize the test slot of the sampler and keep it up-to-date with the “counterfactual” predictor using the sampler$setTestPredictor function. Models that instead modify the response variable can use the sampler$setResponse or sampler$setOffset functions.

One complication in updating the predictor matrix while the sampler runs is that new values of zz must leave the sampler in an internally consistent state. During warmup this constraint is ignored; after warmup is complete, rejection sampling is used to guarantee that no leaf nodes are empty. This is accomplished by using the forceUpdate argument to setPredictor and checking that the logical response is TRUE.

n_warmup  <- 100
n_samples <- 500
n_total   <- n_warmup + n_samples

# Allocate storage for result.
samples_p   <- rep.int(NA_real_, n_samples)
samples_z   <- matrix(NA_real_, n_samples, n)
samples_mu0 <- matrix(NA_real_, n_samples, n)
samples_mu1 <- matrix(NA_real_, n_samples, n)

library(dbarts, quietly = TRUE)
## dbarts: 'bart' is the modern front door (formerly 'bart2'); 'bartBT' is the BayesTree-style one, with 0.9-x's argument names and defaults. A BayesTree-spelled 'bart' call is forwarded to 'bartBT' with a warning for this release.
# We only need to draw one sample at a time, although for illustrative purposes
# a small degree of thinning is done to the BART component.
control <- dbartsControl(updateState = FALSE, verbose = FALSE,
                         n.burn = 0L, n.samples = 1L, n.thin = 3L,
                         n.chains = 1L)
# We create the sampler with a z vector that contains at least one 1 and one 0,
# so that all of the cut points are set correctly.
sampler <- dbarts(y ~ x + z, data_train, data_test, control = control)

# Sample from prior.
p <- rbeta(1, 1, 1)
z <- rbinom(n, 1, p)

# Ignore result of this sampler call
invisible(sampler$setPredictor(x = z, column = 2, forceUpdate = TRUE))
sampler$setTestPredictor(x = 1 - z, column = 2)
sampler$sampleTreesFromPrior()

for (i in seq_len(n_total)) {
  # Draw a single sample from the posterior of f and sigma^2.
  samples <- sampler$run()

  # Recover f(x, 1) and f(x, 0).
  mu0 <- ifelse(z == 0, samples$train[,1], samples$test[,1])
  mu1 <- ifelse(z == 1, samples$train[,1], samples$test[,1])
  
  p0 <- dnorm(y, mu0, samples$sigma[1]) * (1 - p)
  p1 <- dnorm(y, mu1, samples$sigma[1]) * p
  p.z <- p1 / (p0 + p1)
  z <- rbinom(n, 1, p.z)
  
  if (i <= n_warmup) {
    sampler$setPredictor(x = z, column = 2, forceUpdate = TRUE)
  } else while (sampler$setPredictor(x = z, column = 2) == FALSE) {
    z <- rbinom(n, 1, p.z)
  }
  sampler$setTestPredictor(x = 1 - z, column = 2)
  
  n1 <- sum(z); n0 <- n - n1
  p <- rbeta(1, 1 + n1, 1 + n0)

  # Store samples if no longer warming up.
  if (i > n_warmup) {
    offset <- i - n_warmup
    samples_p[offset] <- p
    samples_z[offset,] <- z
    samples_mu0[offset,] <- mu0
    samples_mu1[offset,] <- mu1
  }
}

Saveing the sampler

To use save and load on the sampler, it must first be told to write its state out as an R object:

sampler$storeState()

Visualizing the Result

Finally, we visualize the results. In the graph below, the estimated label of observations encircles the true labels, both of which are represented by point color. We see that, beyond the range of the observed data, the estimated functions regress towards each other and points start to be mislabeled.

mean_mu0 <- apply(samples_mu0, 2, mean)
mean_mu1 <- apply(samples_mu1, 2, mean)

ub_mu0 <- apply(samples_mu0, 2, quantile, 0.975)
lb_mu0 <- apply(samples_mu0, 2, quantile, 0.025)

ub_mu1 <- apply(samples_mu1, 2, quantile, 0.975)
lb_mu1 <- apply(samples_mu1, 2, quantile, 0.025)

curve(f0(x), 0, 80, ylim = c(80, 110), ylab = expression(f(x)),
      main = "Mixture Model")
curve(f1(x), add = TRUE)

points(x, y, pch = 20, col = ifelse(z.0 == 0, "black", "gray"))

lines(sort(x), mean_mu0[order(x)], col = "red")
lines(sort(x), mean_mu1[order(x)], col = "red")

# Add point-wise confidence intervals.
lines(sort(x), ub_mu0[order(x)], col = "gray", lty = 2)
lines(sort(x), lb_mu0[order(x)], col = "gray", lty = 2)

lines(sort(x), ub_mu1[order(x)], col = "gray", lty = 3)
lines(sort(x), lb_mu1[order(x)], col = "gray", lty = 3)

# Without constraining z for an observation to 0/1, its interpretation may be
# flipped from that which generated the data.
mean_z <- ifelse(apply(samples_z, 2, mean) <= 0.5, 0L, 1L)
if (mean(mean_z != z.0) > 0.5) mean_z <- 1 - mean_z

points(x, y, pch = 1, col = ifelse(mean_z == 0, "black", "gray"))
legend("topright", c("true func", "est func", "group 0", "group 1",
                     "est group 0", "est group 1"),
       lty = c(1, 1, NA, NA, NA, NA), pch = c(NA, NA, 20, 20, 1, 1),
       col = c("black", "red", "black", "gray", "black", "gray"),
       cex = 0.8, box.col = "white", bg = "white")

Several Forests with Multipliers

The sampler above holds one function. A model may want several, each entering the response through a multiplier the outer loop knows:

Yi∼N(∑f=1Kmf,igf(xi),σ2),Y_i \sim \mathrm{N}\left(\sum_{f=1}^{K} m_{f,i}\, g_f(x_i),\ \sigma^2\right),

where g1,…,gKg_1, \ldots, g_K are separate sums of trees and mf,im_{f,i} is a known per-observation scalar - an exposure, a dose, a treatment indicator, a group loading. Bayesian causal forests is the K=2K = 2 case with m1≡1m_1 \equiv 1 and m2=zm_2 = z.

One dbartsSampler per forest composes this in R, and the composition is ordinary backfitting once the multiplier is folded in. Given the residual ri=yi−∑g≠fmg,igg(xi)r_i = y_i - \sum_{g \neq f} m_{g,i} g_g(x_i), forest ff’s leaf conditionals want response ri/mf,ir_i / m_{f,i} and precision weight wimf,i2w_i m_{f,i}^2; those are the two channels the loop writes, with setSigma for the third and run(0L, 1L) to advance one sweep. updateScale = FALSE (the default) is what makes the swap coherent: every sweep’s residual is read on the scale creation fixed, so the leaf prior keeps meaning what it meant.

A multiplier at or near zero says the row carries no information about that forest. Divide anyway and the response is amplified without bound, so the recipe snaps such a row to an exactly zero response paired with its exactly zero weight - which is what the engine does internally, at .𝙼𝚊𝚌𝚑𝚒𝚗𝚎$𝚍𝚘𝚞𝚋𝚕𝚎.𝚎𝚙𝚜\sqrt{\texttt{.Machine\$double.eps}}.

set.seed(919)
n.mf <- 200L
x.mf <- matrix(runif(n.mf * 3L), n.mf, 3L)
z.mf <- rbinom(n.mf, 1L, 0.5)
g1.true <- 2 * sin(pi * x.mf[, 1L])
g2.true <- 1 + 2 * x.mf[, 2L]
y.mf <- g1.true + z.mf * g2.true + rnorm(n.mf, 0, 0.5)

# the known per-observation multipliers, one vector per forest
m <- list(rep(1, n.mf), as.double(z.mf))
K <- length(m)

control.mf <- dbartsControl(n.chains = 1L, n.threads = 1L,
                            updateState = FALSE)
# every sampler is built on the SAME response, with k fixed and sigma owned
# by the outer loop
samplers <- lapply(seq_len(K), function(f)
    dbarts(x.mf, y.mf, control = control.mf, leaf.prior = normal(k = 1),
           family = gaussian(sigma = fixed(1)), n.trees = 25L))
# one prior budget divided among the forests, rather than K copies of the
# budget for the whole of the sum
base.mf <- samplers[[1L]]$getLeafPrior()$k.scale / samplers[[1L]]$getK()[1L]
for (s in samplers) s$setLeafPrior(normal(sd = base.mf / sqrt(K)))

g <- rep(list(rep(0, n.mf)), K)
sigma.mf <- 1
n.keep <- 200L
keep.g2 <- matrix(0, n.mf, n.keep)

for (i in seq_len(100L + n.keep)) {
  for (f in seq_len(K)) {
    r <- y.mf - Reduce(`+`, Map(`*`, m[-f], g[-f]))
    zero <- abs(m[[f]]) < sqrt(.Machine$double.eps)
    samplers[[f]]$setResponse(ifelse(zero, 0, r / ifelse(zero, 1, m[[f]])),
                              updateScale = FALSE)
    samplers[[f]]$setWeights(m[[f]]^2)
    samplers[[f]]$setSigma(sigma.mf)
    samplers[[f]]$run(0L, 1L)
    g[[f]] <- as.numeric(samplers[[f]]$getFitsWithoutOffset())
  }
  # sigma is drawn in R, against the COMBINED residual
  resid.mf <- y.mf - Reduce(`+`, Map(`*`, m, g))
  sigma.mf <- sqrt(sum(resid.mf^2) / rchisq(1L, n.mf))
  if (i > 100L) keep.g2[, i - 100L] <- g[[2L]]
}

c(recovery = cor(rowMeans(keep.g2), g2.true), sigma = sigma.mf, truth = 0.5)
##  recovery     sigma     truth 
## 0.9056196 0.4290352 0.5000000

What it costs

Measured on one machine against a single batched engine sampler at matched total trees - the denominator matters, and a price quoted without one says nothing - K samplers driven a sweep at a time cost about 1.9-2.1 times that sampler at K=2K = 2 and 4.6-5.2 times at K=8K = 8, growing roughly linearly in KK: each sampler pays its own per-sweep response install, weight install and result allocation over the full data. Driving one sampler a sweep at a time rather than in a batch is a separate and much smaller charge, about 1.1x.

Five things the recipe does not get

  • No joint move on the multipliers. If the outer loop also draws the mfm_f, it draws them in a block alternating with the forests. A sampler that holds both can interweave them instead, and the two mix differently whenever a forest and its multiplier are jointly weakly identified.
  • The division is yours. The caller forms r/mfr / m_f and so owns the conditioning, including the snap above.
  • KK quantized predictor stores and KK sampler states, where one sampler holding KK forests keeps one of each.
  • KK leaf calibrations to state, one per sampler, rather than a single calibration stated across the forests at creation.
  • A serial outer loop. The forests are drawn one after another in R, so the composition cannot spread them across threads.

Graduating to a multi-forest sampler

dbarts(..., forests = list(forest(), forest(basis = factor(z)))) fits this model in one sampler, and the question a prototype should be able to ask is whether moving to it changes the posterior being targeted.

The additive decomposition ports exactly. What does not port by itself is prior shape. The two routes agree only if the caller

  1. centers yy at its midrange, or absorbs the response transform’s shift with setOffset: each single-forest sampler’s reported fit carries that shift, so the composition attributes it once per forest where the sampler attributes it once;
  2. pins the leaf prior identically - leaf.prior = normal(k) with kk fixed, every sampler built on a common yy, and updateScale = FALSE throughout;
  3. accepts that with a multiplier far from 1 the internal response a forest sees leaves the band its leaf prior was calibrated for.

Without (1) and (2) the two routes are different priors that agree closely. With them the remaining difference is the leaf calibration alone. Measured on a pinned two-forest sampler against the recipe above, at n=400n = 400 over eight chains: with the remedies the posterior mean conditional effect agreed to 3.4% pointwise, inside Monte Carlo error in root-mean-square; without them the difference was 11.9% pointwise and several times Monte Carlo error. Both recovered the truth about equally well, which is the point - the difference is a prior, not a defect.

State equally plainly what the recipe does not reach and never will: a response family, link or latent structure the sampler does not provide, and a leaf or hyperprior it does not provide. Those are selected by argument, not composed in R. forest() carries the same scale knobs the composition sets by hand, and $setForestBasis, $setForestWeights and $getForestAmplitudes are the post-creation routes into a multi-forest sampler, so a prototype ports without changing the model it targets.

Response Families

Everything above fits a continuous response, dbarts’s default (family = "auto", which chooses "gaussian" for a continuous yy and "probit" for one coded 0/1). family can also be set to "gaussian", "probit", or "logistic" explicitly. The two binary families model P(Y=1∣x)=F(f(x))P(Y = 1 \mid x) = F(f(x)) for a link FF and the sum-of-trees ff, and report fits and predictions on that latent scale rather than as probabilities (pnorm/plogis recovers the probability from it). Probit takes FF to be the standard normal CDF; logistic uses Polya-Gamma augmentation and takes FF to be the logistic CDF, whose latent variable has heavier tails than probit’s, so the leaf-value prior widens to match (its scale becomes pi * sqrt(3) in place of probit’s 3), without naming the slot.

set.seed(11)
n.fam <- 60
x.fam <- runif(n.fam)
y.fam <- rbinom(n.fam, 1, plogis(3 * (x.fam - 0.5)))

fit.logistic <- dbarts(y.fam ~ x.fam, family = "logistic",
                       control = dbartsControl(n.chains = 1L,
                                                n.threads = 1L), n.trees = 10L)
samples.logistic <- fit.logistic$run(20L, 20L)

DART

When only a handful of predictors out of many are relevant, the CGM tree prior’s uniform choice of split variable wastes proposals on the rest. split.probs = dart(...) replaces it with a Dirichlet prior over split-variable probabilities (Linero 2018): the sampler learns which variables matter and increasingly proposes splits on them, which doubles as a variable-importance diagnostic.

set.seed(12)
n.dart <- 60
x.dart <- matrix(runif(n.dart * 8), n.dart)
y.dart <- 4 * x.dart[, 1] + rnorm(n.dart, 0, 0.2)

fit.dart <- dbarts(y.dart ~ x.dart, split.probs = dart(),
                   control = dbartsControl(n.burn = 100L,
                                            n.chains = 1L, n.threads = 1L),
                                            n.trees = 10L)
samples.dart <- fit.dart$run(100L, 30L)

# One row per predictor, averaged over the retained samples: only the first
# column of x.dart drives y.dart, and its split probability dominates.
round(rowMeans(samples.dart$varprobs), 3)
## [1] 0.991 0.000 0.005 0.001 0.000 0.000 0.001 0.002

The Dirichlet concentration alpha is itself sampled by default (update.alpha = TRUE); a smaller alpha concentrates the split probabilities more aggressively. ?dbartsPriors documents the rest of dart’s arguments.

Data Frames, Factors, and Missing Values

dbarts and bart take a data.frame directly, and (factors = "categorical", the default for both) an unordered factor predictor becomes a single categorical column: its splits send a subset of levels down each branch, rather than the one-indicator-per-level expansion bartBT still uses. factors = "indicators" recovers that expansion on either function, useful for comparing the two. Ordered factors are their own predictor kind regardless of factors: coded on their integer levels and split at a threshold, with one candidate threshold per level boundary rather than a fixed count of cuts, so n.cuts does not reach them.

Missing predictor values need not be removed beforehand. By default every split rule learns a direction for NA: an observation whose split variable is missing follows that rule’s direction (“Missing Incorporated in Attributes”, Twala et al. 2008), estimated jointly with the rest of the tree. na.action = na.fail instead refuses any missing value, the response’s too.

set.seed(13)
n.miss <- 60
g <- factor(sample(c("a", "b", "c"), n.miss, replace = TRUE))
x.miss <- rnorm(n.miss)
x.miss[sample(n.miss, 10)] <- NA
y.miss <- ifelse(g == "a", 2, -1) +
    ifelse(is.na(x.miss), 0, x.miss) + rnorm(n.miss, 0, 0.2)

df.miss <- data.frame(y = y.miss, x = x.miss, g = g)
fit.miss <- bart(y ~ x + g, df.miss, n.trees = 10L, n.chains = 1L,
                  n.threads = 1L, n.samples = 20L, n.burn = 20L,
                  verbose = FALSE)

g above ingests as one categorical predictor with three levels rather than two indicator columns, and the ten missing values of x are incorporated rather than dropped.

A Random-Intercept Model Through setOffset

The response-side conduit carries a whole outer block just as setPredictor carries the predictor side. A random-intercept model yi=f(xi)+bg(i)+ϵi,ϵi∼N(0,σ2),bj∼N(0,τ2) y_i = f(x_i) + b_{g(i)} + \epsilon_i, \quad \epsilon_i \sim N(0, \sigma^2), \quad b_j \sim N(0, \tau^2) is the smallest worked example: bb and τ\tau are two Gibbs blocks the outer loop owns, and BART sees them only as an offset. Each sweep draws bb from its conjugate normal against the current residual, installs ff’s new offset with setOffset, runs one sweep, subtracts the offset back off the reported fit, and then draws τ\tau.

setOffset(offset, updateScale) takes updateScale = FALSE after warmup: the response transform BART calibrated at creation must stay put, or bb and τ\tau - which the outer loop holds in response units - would come to mean something else from one sweep to the next.

set.seed(24)
n_ri     <- 200L
n_groups <- 8L
g        <- rep_len(seq_len(n_groups), n_ri)
x_ri     <- matrix(runif(n_ri * 3L), n_ri, 3L)
b_true   <- rnorm(n_groups, 0, 0.8)
y_ri     <- sin(pi * x_ri[, 1L]) + 2 * x_ri[, 2L] + b_true[g] + rnorm(n_ri, 0, 0.5)

n_warmup_ri  <- 200L
n_samples_ri <- 500L

control_ri <- dbartsControl(updateState = FALSE, verbose = FALSE,
                            n.burn = 0L, n.samples = 1L, n.chains = 1L,
                            n.threads = 1L)
sampler_ri <- dbarts(x_ri, y_ri, control = control_ri, n.trees = 50L)

# rows of each group, and its size; the conjugate draw needs both
rows_of  <- split(seq_len(n_ri), g)
size_of  <- lengths(rows_of)

# Half-Cauchy(0, 2.5 sd(y)) on tau, sampled by a random-walk Metropolis step
tau_scale <- 2.5 * sd(y_ri)
log_tau_posterior <- function(tau, b) {
  if (tau <= 0) return(-Inf)
  -n_groups * log(tau) - 0.5 * sum(b^2) / tau^2 +
    dcauchy(tau, 0, tau_scale, log = TRUE)
}

b     <- rnorm(n_groups, 0, tau_scale / 5)
tau   <- tau_scale / 5
sigma <- sd(y_ri)
f_hat <- rep_len(0, n_ri)

samples_b   <- matrix(NA_real_, n_samples_ri, n_groups)
samples_tau <- rep_len(NA_real_, n_samples_ri)

for (i in seq_len(n_warmup_ri + n_samples_ri)) {
  # b | f, sigma, tau: independent normals, one per group
  resid     <- y_ri - f_hat
  post_var  <- 1 / (size_of / sigma^2 + 1 / tau^2)
  post_mean <- post_var * (vapply(rows_of, function(rows) sum(resid[rows]), 0) /
                             sigma^2)
  b <- rnorm(n_groups, post_mean, sqrt(post_var))

  # f, sigma | b: one BART sweep against the intercepts as an offset. The
  # scale re-anchors during warmup only, and is pinned for every kept draw.
  sampler_ri$setOffset(b[g], updateScale = i <= n_warmup_ri)
  draw  <- sampler_ri$run()
  f_hat <- as.vector(draw$train) - b[g]
  sigma <- draw$sigma[1L]

  # tau | b: a random-walk Metropolis step on the log scale
  proposal <- tau * exp(rnorm(1, 0, 0.2))
  logRatio <- log_tau_posterior(proposal, b) - log_tau_posterior(tau, b) +
    log(proposal) - log(tau)
  if (log(runif(1)) < logRatio) tau <- proposal

  if (i > n_warmup_ri) {
    samples_b[i - n_warmup_ri, ] <- b
    samples_tau[i - n_warmup_ri] <- tau
  }
}

# the group intercepts and their scale are only identified up to a shared
# level, which the forest absorbs, so compare them centered
cor(colMeans(samples_b) - mean(colMeans(samples_b)), b_true - mean(b_true))
## [1] 0.99589

The same shape carries any outer block whose contribution to the linear predictor is a per-observation vector: a parametric term, a spatial effect, a frailty. setResponse is the companion for a block that changes yy itself rather than shifting it.

Multiple Threading

When implementing a Gibbs sampler using dbarts in R, it is most often the case that multiple threads will need to be handled by creating separate copies of the sampler and data. While dbartsSamplers are natively multithreaded, few of their slots are stored independently across chains, so an outer loop that owns per-chain state runs one single-chain sampler per chain and threads them itself.

This custom-loop bookkeeping is only needed when embedding BART in a larger sampler. A plain multi-chain bart fit already tracks its chain dimension: summary() reports split-R-hat and effective sample size for the scalar parameters the fit sampled (sigma, and k when it is drawn) and names any it held fixed on a line under the table, and dbarts::extract(fit, "sigma", combineChains = FALSE) returns a sampled parameter’s draws chain-separated, as a chains-by-samples matrix; a parameter held fixed comes back as its one number.