Crossvalidation For Bayesian Additive Regression Trees
xbart.RdFits the BART model against varying k (or sd), power, base, and n.trees parameters using \(K\)-fold or repeated random subsampling crossvalidation, sharing burn-in between parameter settings. Results are returned as an array of evaluations of a loss function on the held-out sets.
Usage
xbart(
formula, data, subset, weights, offset, verbose = FALSE, n.samples = 200L,
method = c("k-fold", "random subsample"), n.test = c(5, 0.2),
n.reps = 40L, n.burn = c(200L, 150L),
loss = c("rmse", "log", "mcr"), n.threads = dbarts::guessNumCores(), n.trees = 75L,
k = NULL, sd = NULL, power = 2, base = 0.95,
split.probs = NULL, drop = TRUE,
sigest = NULL,
seed = NULL,
factors = c("categorical", "indicators"),
family = c("auto", "gaussian", "probit", "logistic"),
weights.type = NULL,
leaf.prior = NULL, n.cuts = 100L, useQuantiles = FALSE, n.thin = 1L,
storage = c("double", "single"),
parallel = getOption("dbarts.parallel", "auto"), cl = NULL,
control = dbarts::dbartsControl(), sigma = NULL, ...)Arguments
- sigma
The 0.9-x spelling of
sigest, accepted for one release with a once-per-session warning and removed in dbarts 1.1-0. Supplying both is an error.- ...
Not used for new code: the channel that lets a retired argument spelling (
resid.prior, nowfamily = gaussian(sigma = ), its one home, so writing it both ways is refused where the two disagree) reach a message naming its successor, instead of R's own “unused argument” error. Any other name is refused. Removed in dbarts 1.1-0.- control
A
dbartsControlobject carrying the sampler and engine settings, including the onesxbartspells no flat name for:categoricalExhaustiveCap,testFitParallelCutoff,predictParallelCutoff,sparseDensityThreshold,treeShift, and the tree-move mixtureproposal.probs. Precedence, one rule: a flat argument named in the call wins over the control's slot of the same name, and a slot the control speaks for - one its owndbartsControl()call named, or one edited afterwards to differ from a fresh control's - wins over this function's default, while a slot it never spoke for leaves that default standing. The fields a sweep forces -n.chains, the control's ownn.threads,keepTrees,keepTrainingFits,updateState,verbose- are applied after both, andn.burnandn.threadshere are the sweep's own grid axis and worker count rather than the control fields of the same name. A control taken from a fitted sampler carries that fit's run settings and nothing else: it sets no tree count, cut count or quantile rule for this fit, and one that differs from what the call takes draws a warning saying it is not used.- formula
An object of class
formulafollowing an analogous model description syntax aslm. For backwards compatibility, can also be thebartBTmatrixx.train. Seedbarts. AdbartsDataobject carryingbases, the data of a sampler of several forests, is cross-validated as a model of one forest, the bases left out, asdbartsreads it given noforests:xbartcross-validates single-forest models.- data
An optional data frame, list, or environment containing predictors to be used with the model. For backwards compatibility, can also be the
bartBTvectory.train.- subset
An optional vector specifying a subset of observations to be used in the fitting process.
- weights
An optional vector of weights to be used in the fitting process. For a gaussian response, BART fits a model with observations \(y \mid x \sim N(f(x), \sigma^2 / w)\), where \(f(x)\) is the unknown function. Binary responses differ: a
"probit"model does not support weights (a weighted probit has no tractable latent-variable form), except that weights identically 1 are treated as absent and weights all 0 and 1 name the rows in the data set, as indbarts: a fold's training rows of weight 0 are out of its likelihood and its held-out rows of weight 0 out of its loss, as on every family; a"logistic"model treats them as observation counts and so requires non-negative whole numbers, a 0 out of the likelihood (its Polya-Gamma latent for a count \(w\) is a sum of \(w\) unit draws, so a sweep's time grows with the total count, and a count above \(10^6\) is refused). A gaussian row of weight 0 takes no part in a fold's response scale or starting sigma: each fold reads the rows of positive weight among its training rows, and where those hold fewer than two distinct values of the response it reads every training row, one warning of classdbartsScaleFallbackWarningbeing raised for the whole call when that is so of all rows.- offset
An optional vector specifying an offset from 0 for the relationship between the underlying function, \(f(x)\), and the response \(y\). Useful for all three response families, though each applies it differently: for a gaussian response, \(y = f(x) + \mathrm{offset} + \epsilon\), a fixed component of the mean (see
dbartsfor the interaction with BART's internal range-scaling); for binary responses it enters the link directly, \(P(Y = 1 \mid X = x) = \Phi(f(x) + \mathrm{offset})\) for a"probit"fit and the analogous model on the logistic link for"logistic".- verbose
A logical determining if additional output is printed to the console, including the one-line message
family = "auto"prints, once per call, naming the family it resolves to.- n.samples
A positive integer, setting the number of posterior samples drawn for each fit of training data and used by the loss function.
- method
Character string, either
"k-fold"or"random subsample".- n.test
For each fit, the test sample size or proportion. For method
"k-fold", is expected to be the number of folds, and in \([2, n]\). For method"random subsample", can be a real number in \((0, 1)\) or a positive integer in \((1, n)\). When a given as proportion, the number of test observations used is the proportion times the sample size rounded to the nearest integer.- n.reps
A positive integer setting the number of cross validation steps that will be taken. For
"k-fold", each replication corresponds to fitting each of the \(K\) folds in turn, while for"random subsample"a replication is a single fit.- n.burn
Unlike the single-scalar
n.burnofdbartsandbart, here it is one or two non-negative integers, specifying 1) the burn-in when a chain is freshly started against a data split and 2) the burn-in when moving from one parameter setting to another over the same split. A longer vector is an error naming the argument rather than being silently truncated to its first two entries; dbarts 0.9-x read a third element as a per-replication burn-in, which no longer exists. Chains are never carried between data splits or folds - the held-out observations of one were training observations of the previous, so continuing a chain lets slowly-mixing settings score against data they have effectively seen.- loss
Either one of the pre-set loss functions as character-strings (the default is
"rmse"for a continuous response and"log"for a binary one;mcr- misclassification rate for binary responses,rmse- root-mean-squared-error for continuous response),log- negative log-loss for binary response (rmseserves this purpose for continuous responses), a function, or a function-evaluation environment list-pair. A function keeps its own environment, so a closure reads what it captured; the list form calls the function from the given environment. Functions should have prototypes of the formfunction(y.test, y.test.hat, weights), wherey.testis the held out test subsample,y.test.hatis a matrix of dimensionlength(y.test)\(\times\)n.samples, andweightsare an optional vector of user-supplied weights. See examples.- n.threads
Every (replication, fold) pair is an independent unit of work, and for
n.threads > 1the units are divided into approximately equal chunks and executed on that many parallel workers (seeparallel). Ak-fold run of a single replication therefore uses up tokworkers. The default usesguessNumCores, which should work across the most common operating system/hardware pairs, and is one where the cores cannot be counted; a statedNAis refused. Warnings raised while fitting or scoring, including by a suppliedlossfunction, are collected from every unit and signaled in the calling session once all units have finished, in unit order and each distinct warning once, so the same warnings are seen at anyn.threads.- parallel
One of
"auto","fork"or"socket", and by defaultgetOption("dbarts.parallel", "auto")."fork"runs the workers as forked copies of the calling session, which is fast to start and shares the data;"socket"starts separate R sessions (amakeClustercluster) that each load dbarts and receive their own copy of the data, which costs roughly 80 MB per worker beyond the data itself."auto"forks unless the platform is Windows, the session runs under RStudio or Positron, or R.app, in which case it uses sockets."fork"is an error on Windows. Forking can hang or crash when the calling session has already used a library that is not fork-safe, such as a multithreaded BLAS; if so, setparallel = "socket"oroptions(dbarts.parallel = "socket"). Results do not depend on the kind of worker. Ignored whenclis given or only one worker is used.- cl
NULLor a cluster from theparallelpackage that the caller has already started, used instead of starting workers whateverparallelsays. The units are divided intomin(n.threads, length(cl))chunks, andxbartdoes not stop the cluster. The workers must load the same dbarts version as the calling session, since the work function is sent by reference and each worker runs its own installed dbarts. A one-nodecl, liken.threads = 1, runs the units in the calling session.- n.trees
A vector of positive integers setting the BART hyperparameter for the number of trees in the sum-of-trees formulation. See
bart. A writtenNULLis refused, saying to leave the argument out for its default.- k
The grid for the BART hyperparameter setting the leaf-mean prior standard deviation: a vector of positive real numbers, each a cell of its own, or a
listwhose entries are positive numbers andkhyperpriors (dbartsPriors), so thatk = list(1, 2, chi())sweeps two fixed values against the modeled one. A modeled cell holds its hyperprior and DRAWSkevery sweep, so its loss is computed under a shrinkage that moves within the fit rather than under a named value; it is labeled in the result by the constructor call that rebuilds it. IfNULL, one cell is run at the defaultbartwould fit for the response type: fixed 2 for a continuous response andchi(1.5, 2)for a binary one. Akcarried by a suppliedleaf.priorstands in for a missing argument. Fixed cells are always swept largest to smallest, with a modeled cell last, and the reportedkaxis is un-permuted back to the order given, so results do not depend on the orderkis listed in; cells still warm-start off the previous one, so this is order-invariance, not an unbiased estimate for each cell taken alone. A writtenNULLis the argument left out.- sd
The same grid axis stated in absolute spreads instead, exclusive with
k: a vector of leaf-prior sds on the family's scale (seenormal(sd = )indbartsPriors), or alistmixing them withinvchipriors. Each spread is held fixed across folds, where akis relative to each fold's own training range; on a binary family, whosek.scaleis a constant of the latent scale, a grid of fixed cells is the same sweep either way (sd = k.scale / k); a modeled cell is not, sinceinvchi(df, c)starts its chain at the spreadcwherechi(df, k.scale / c)starts atk.scale / 2. Those starts are the first cell's: each later cell of the warm sweep starts at the spread the previous cell ended on, whichever ofkandsdthe grid is stated in. Akinsideleaf.priorbeside ansdgrid is refused, as ansdthere is beside either grid. The cells are swept most shrunk first - the smallest sd - with the warm starts and unit parallelism thekgrid has, and the result's axis is labeledsd.- power
A vector of positive real numbers, setting the BART hyperparameter for the tree prior's growth probability, given by \({base} / (1 + depth)^{power}\). A written
NULLis refused, saying to leave the argument out for its default.- base
A vector of real numbers in \((0, 1)\), setting the BART hyperparameter for the tree prior's growth probability. A written
NULLis refused, saying to leave the argument out for its default.- split.probs
Prior probabilities that a variable is used in a splitting rule, as in
bart. A single value orNULLyields the uniform default; a named or unnamed vector assigns per-column probabilities;num.varsstands for the number of predictors; anddart()fromdbartsPriorsruns DART (Linero 2018) in every cell. Fixed for the whole crossvalidation, not part of the swept grid.xbarttakes notree.prior:powerandbaseare grid axes, and given atree.priorit refuses, naming the plain spelling.- drop
Logical, determining if dimensions with a single value are dropped from the result.
- sigest
A positive numeric estimate of the residual standard deviation; one of
"auto","dense"and"sparse", naming how one is made; or a function that makes one. Each name gives the residual standard deviation of the linear regression of the response on the predictors, factors entering as indicator columns whatever the trees are given, and, where that regression leaves no residual degrees of freedom, counting each row once, not as often as its weight, under weights read as counts, the standard deviation of the response over the observations of positive weight, weighted as the regression is (summary(lm(y ~ 1, weights = weights))$sigma,sd(y)at unit weights; for weights read as counts, the standard deviation of the rows written out, orsd(y)over the observations of positive weight where the counts total at most 1, up to the rounding of their sum), which a line underverbosesays, with no warning."auto"islm's own routine ("dense") for a matrix or a data frame and an exact sparse one ("sparse") where the caller supplied a sparse-stored column.NULL(the default) is"auto"; an explicitNAis a missing value and is refused. The estimate is made separately for each fold on that fold's training rows, so no fold's prior reads its held-out responses; it is made on all rows once beforehand, to raise any refusal once, to print its lines underverboseand to settle the routine every fold runs, a fold taking its own response's standard deviation where all rows, or its own, leave no residual degrees of freedom; where all rows allow the regression and some folds' rows do not, one more line underverbosegives, for each standard deviation taken, the number of (replication, fold) units that took it. A function is called once for each fold in place of any of that, asfunction(x, y, weights, offset)or inloss's list form with an environment, on the worker that runs the fold and so under whatlossandseedsay of workers and random numbers.xis the fold's training rows of the design"auto"would read, a numeric matrix or for a sparsexadgCMatrix, built only if the function reads it;y,weightsandoffsetare those rows',NULLwhere the call has none; the value must be one positive finite number.function(x, y, weights, offset) sd(y)skips an expensive estimate, and the alternativesdbartslists for one can be written the same way. Every entry point spells thissigest-xbart,bartBT/bartand the sampler constructorsdbarts/dbartsSpecalike;sigmais the retired 0.9-x spelling ondbartsand onxbartitself (dbartsSpecrefuses it); a family object's ownsigmais the prior this estimate calibrates, not the estimate. It is the estimate achisqresidual prior's quantile is calibrated against, so it stands besidefamily = gaussian(sigma = chisq(df, quant))and has nothing to calibrate besidefamily = gaussian(sigma = fixed(value)), which fixes the residual scale outright: no estimate is made there, by any rule, and a number that differs from the square root ofvalue, or a function, is refused.- seed
Optional integer specifying the desired pRNG seed.
NULL(the default) means not given here and defers to a seed already sitting incontrol, if any; anNAis a missing value, as inset.seed: it reads asNULLfor one release, with a once-per-session warning. From the seed in force,xbartdraws a split seed for each replication and a seed for each sampler its units create - one per (replication, fold) unit at each distinct tree count - withsample.int, in one pass, under the caller's ownRNGkind: a given seed therefore draws different values, and so gives different results, under a different kind. Each unit's sampler then takes its seed throughcontrol's seed slot, exactly asbart's ownseeddoes, driving a dedicated generator that never reads R's stream, so results are reproducible at anyn.threadsfor a given seed andRNGkind. A suppliedlossfunction that draws random numbers draws on the worker running it, from that process's own default generator; atn.threadsgreater than 1 this makes the run not reproducible, whatever seedxbartitself is given. The caller's random stream is left untouched when a seed is given, and advanced only by that derivation when one is not; without a seed,set.seedbeforehand suffices. See the Reproducibility section of bart.- factors
How factor columns in a data frame enter the model; as in
dbarts.- family
The response model, resolved as in
dbarts:"auto"fits gaussian models to continuous responses and probit models to those coded 0/1,"gaussian"forces a continuous fit, and"probit"and"logistic"require a 0/1 response,family = binomialor"logistic"also taking a binomial response,cbind(successes, failures), asbartdoes. A two-level factor, logical, or two-level character response is detected and fit as probit; a factor with three or more levels is an error, asxbartdoes not cross-validate the multinomial model, and neither is another matrix response (per-category counts or a (time, status) pair), which is refused namingbart. A survival response - aSurvobject or two-column(time, status)pair, onformulaor asdatadirectly - is likewise an error namingbart/dbartswithfamily = "aft"/"hazard":xbartdoes not cross-validate a survival model. The built-in binary losses transform test predictions through the family's link. This vocabulary is narrower thanbart's by design -xbartcross-validates a single scalar loss per fold, which the own-class families' K-forest or two-part fits have no single counterpart of; the wider family set lives onbart. Base R family objects map asglmtakes them (binomialis the logit link, sofamily = binomialis"logistic", not probit) and the rest are refused; seedbartsFamilies.- weights.type
How
weightsare read, as indbarts:NULL(the default) is precision on a gaussian fit and frequency on a probit or logistic one,"precision"is refused on those two, and"frequency"on a gaussian fit reads a row of weight \(w\) as \(w\) identical observations, each fold's starting sigma being the residual sd of the regression on its training rows written out that many times.- leaf.prior
An optional expression of the form
normal(k),linear(columns, k), orgp(columns, k, ...)selecting the leaf model, as indbarts. The default fits constant leaves. Akgiven inside the prior stands in for a missingkargument; thekargument otherwise drives the crossvalidation grid as usual. Each fold centers and scales alinearorgpleaf's covariates over its own training rows of positive weight, as a fit on those rows would. Where those rows hold fewer than two distinct values of a gaussian response, the fold reads its response's scale and the covariates over every training row; otherwise a fold whose such rows hold one value of a covariate, or values that differ by rounding only, refuses the call, unless itsgpleaf stateslengthscale. A writtenNULLis the argument left out.A named
sdinside the prior (normal(sd = ), and likewise forlinearandgp) stands in for a missingsdargument, as a one-cellsdaxis. Beside akorsdgrid it is refused, since both would state the spread: name the spreads in thesdgrid. Two calls with the sameseeduse the same folds whatever their spreads, so one call persdreproduces each cell a grid's fresh start would.- n.cuts
The number of decision rules used for each predictor, as
bart'sn.cutsreads: one positive integer for every column, one per column, or by column or term name with.defaultfor the rest. AdbartsDataobject given asformulakeeps its own, and a grid it carries, itscutPoints, is the one every fold takes, asdbartstakes it; statingn.cutsoruseQuantilesderives the grid again.- useQuantiles
Logical; as in
bart, determines whether decision rules are placed at the empirical quantiles of each predictor's values rather than spaced uniformly through its range. Either way a fold takes the full data's rules and their weights, so on a predictor with fewer distinct values thann.cutsit holds a rule between held-out values as well.- n.thin
A positive integer; as in
dbartsControl, thins each cell's chain against serial correlation.n.samplesare still returned regardless.- storage
A character string selecting the precision of the internal running residual, spelled and defaulted as in
dbartsControl. Every cell's sampler is created over a shared per-fold data handle, a path the engine keeps in double precision regardless of family or leaf model;"single"is refused rather than silently ignored.
Details
A fit in which every predictor holds one value on the rows kept, with no forest having a basis, is refused, naming the columns, as by bart.
Crossvalidates n.reps replications against the crossproduct of given hyperparameter vectors n.trees \(\times\) k \(\times\) power \(\times\) base. For each fit, either one fold is withheld as test data and n.test - 1 folds are used as training data or n * n.test observations are withheld as test data and n * (1 - n.test) used as training. A replication corresponds to fitting all \(K\) folds in "k-fold" crossvalidation or a single fit with "random subsample". The training data is used to fit a model and make predictions on the test data which are used together with the test data itself to evaluate the loss function.
loss functions are either the default of average negative log-loss for binary outcomes and root-mean-squared error for continuous outcomes, misclassification rates for binary outcomes, or a function with arguments y.test, y.test.hat and weights. y.test.hat is of dimensions equal to length(y.test) \(\times\) n.samples, on the latent scale for binary outcomes. For a binary outcome the built-in "rmse" compares the response with the posterior mean probability (the link applied to each draw, then averaged), as "log" and "mcr" do, while a supplied function receives the latent draws. A third option is to pass a list of list(function, evaluationEnvironment), so as to provide default bindings. The binary losses "log" and "mcr" cannot apply to continuous responses. On a binomial response a row is read as its trials: y.test is its observed proportion \(k / n\) (0 at a row of no trials), weights are its trials times its weight, and the built-in losses read each row so weighted, "log" as \(-[y \log p + (1 - y) \log(1 - p)]\) and "mcr" as the share of the row's trials on the wrong side of 0.5, which is what both losses give on the rows written out one trial each; "rmse" compares the proportion with the probability, a row weighted by its trials, and on the rows written out would add a term that depends on the data alone. A supplied function receives the same y.test and weights, and the latent draws.
Value
An array with up to six dimensions, in order rep (length n.reps), n.trees, k (or sd), power, base, and loss. rep is always present. n.trees, k, power, and base are each omitted when drop is TRUE and the corresponding grid has length 1; with drop = FALSE they are always present, an absent k included, whose single cell is then named for the default it ran at. The trailing loss dimension, sized to however many values a single call to loss returns, is present only when that count is greater than one - independent of drop entirely; the default losses and an ordinary scalar-returning custom loss never contribute it. When none of the above survive, the result collapses to a plain vector of length n.reps. When the result remains an array, its dimnames name the swept values on each surviving hyperparameter axis - exact integer labels for n.trees, values rounded to 2 significant digits for the double-valued k/power/base axes, and the constructor call for a modeled k cell; the loss axis, when present, carries no per-slot names, and neither does rep.
For method "k-fold", each element is an average across the \(K\) fits. For "random subsample", each element represents a single fit.
The result is a bare array with no class, so the fit generics - predict, extract, fitted, residuals - do not apply to it; it is a table of losses, not a fit.
Author
Vincent Dorie: vdorie@gmail.com
Examples
f <- function(x) {
10 * sin(pi * x[,1] * x[,2]) + 20 * (x[,3] - 0.5)^2 +
10 * x[,4] + 5 * x[,5]
}
set.seed(99)
sigma <- 1.0
n <- 100
x <- matrix(runif(n * 10), n, 10)
Ey <- f(x)
y <- rnorm(n, Ey, sigma)
mad <- function(y.test, y.test.hat, weights) {
# note, weights are ignored
mean(abs(y.test - apply(y.test.hat, 1L, mean)))
}
## low iteration numbers to to run quickly
xval <- xbart(x, y, n.samples = 15L, n.reps = 4L, n.burn = c(10L, 3L),
n.trees = c(5L, 7L),
k = c(1, 2, 4),
power = c(1.5, 2),
base = c(0.75, 0.8, 0.95), n.threads = 1L,
loss = mad)