Run Mutation Testing for an R Package — mutate

Mutates all .R files under a package's R/ directory, runs the package's tests against each mutant in parallel, and summarizes mutation outcomes.

Usage

mutate_package(
  pkg_dir,
  cores = max(1, parallel::detectCores() - 2),
  isFullLog = FALSE,
  detectEqMutants = FALSE,
  mutation_dir = NULL,
  max_mutants = NULL,
  timeout_seconds = NULL,
  config_dir = getwd(),
  max_line_deletions = 0,
  cran = TRUE,
  fail_fast = TRUE,
  isolate = FALSE,
  exclude_files = NULL,
  strategy = c("auto", "testthat", "tinytest", "tinytest-installed", "installed"),
  coverage_guided = TRUE,
  coverage_backend = c("record_tests", "per_file"),
  target_margin = NULL,
  confidence = 0.95,
  max_show = 50L
)

Arguments

pkg_dir: Path to the package directory.
cores: Number of parallel workers used for mutant test execution.
isFullLog: Logical; if TRUE, prints per-mutant logs and timeout info.
detectEqMutants: Logical; if TRUE, every generated mutant is analyzed for equivalence using the OpenAI-based workflow before the test suites are run. Mutants judged equivalent are recorded as survived without running their tests, as no test can kill an equivalent mutant; the remaining mutants are tested as usual.
mutation_dir: Optional directory to store generated mutant files. If NULL, a temporary directory is used.
max_mutants: Sample that number of mutants for testing. If NULL, all mutants are tested.
timeout_seconds: Optional timeout in seconds for each mutant run. If NULL, timeout is derived from baseline runtime with a small minimum floor. Still works with compiled native code.
config_dir: Directory searched for a .openai_config file when detectEqMutants = TRUE (see get_openai_config()). Defaults to the current working directory.
max_line_deletions: Maximum number of line-deletion mutants per .R file (passed to mutate_file()); 0 disables them. Defaults to 0, since line-deletion mutants are largely redundant with the AST block-deletion mutants generated by default.
cran: Logical; if TRUE (the default), tests run in "CRAN mode": the NOT_CRAN environment variable is set to "false" in the test subprocess so testthat::skip_on_cran() / skip_if_offline() guards take effect and the same tests CRAN would run are used (skipping network/slow tests the package marks). Set to FALSE to run the full suite (NOT_CRAN = "true"), as devtools::test() does.
fail_fast: Logical; if TRUE (the default), a mutant's test run stops at the first failing test rather than running the whole suite. A mutant is KILLED as soon as one test detects it, so the remainder of the suite is wasted work. Set to FALSE to run the full suite for every mutant. Applies to the testthat strategy; the tinytest strategy always runs the full set of selected test files, and the installed-tests fallback already stops at the first failing test file regardless of this flag.
isolate: Logical; if FALSE (the default), each mutant's package copy symlinks the unchanged directories of the original package (only the mutated R/ file is materialised), which is fast but makes those directories shared writable state across the parallel workers. If TRUE, the src/ and tests/ directories (or src/ and inst/ under the tinytest strategy) are deep-copied into every mutant copy instead. Use isolate = TRUE when a package has non-hermetic tests that write files into tests/ (or src/) and parallel runs therefore produce spurious KILLED/HANG verdicts; it gives each worker its own copy at the cost of extra disk. Note that running with cores = 1 avoids such contention without the copy cost.
exclude_files: Optional character vector of shell-style glob patterns (e.g. "import-standalone-*") matched against the base names of the .R files in R/. Matching files are skipped entirely before any mutants are generated. NULL (the default) mutates every file. This complements the in-source # mutator:ignore-file and # mutator:ignore-start / # mutator:ignore-end directives, which exclude a whole file or a line region from within the source itself. Note that for operator mutations the engine only resolves positions to the enclosing top-level definition, so a region directive excludes that function's operator mutants as a group (line-deletion mutants are excluded line-precisely).
strategy: Test strategy to use. "auto" (the default) picks the testthat strategy when tests/testthat/ exists, the tinytest strategy when inst/tinytest/ exists, and the installed-tests strategy otherwise. "testthat" forces the in-process testthat::test_dir() path (requires tests/testthat/). "tinytest" forces the in-process tinytest::run_test_dir() path (requires inst/tinytest/). Both in-process strategies load the package with pkgload::load_all(), which does not dispatch S4 methods defined on ...-dispatching base generics such as seq(); a package that relies on those will fail the baseline, and the error points to "tinytest-installed". "tinytest-installed" runs the tinytest suite against an installed copy (R CMD INSTALL + tinytest::test_package(), requires inst/tinytest/): slower than dev-mode but matches an installed package (correct S4 dispatch) and still supports coverage guidance. "installed" forces the R CMD INSTALL --install-tests + tools::testInstalledPackage() path (requires tests/).
coverage_guided: Logical; if TRUE, only the tests that actually exercise a mutant's mutated line(s) are run for that mutant, instead of the whole suite. Coverage is measured once on the unmutated package with covr. A mutant on a line no test covers cannot be killed, so it is reported SURVIVED without running any test. Selection is at the test-file level; under the assumption that the suite deterministically exercises the code, it should not change a mutant's verdict, only which tests run. Defaults to TRUE. Coverage guidance is available under the testthat, tinytest, and tinytest-installed strategies; when the resolved strategy is the generic installed-tests fallback, mutator emits a warning and runs the full suite for every mutant. Pass FALSE to disable it (and silence that warning).
coverage_backend: How coverage_guided attributes coverage to tests under the testthat strategy (ignored when coverage_guided = FALSE and for the tinytest strategies, which have a single per-file driver). "record_tests" (the default) uses covr's record_tests in a single run; it relies only on covr's public output but, because covr credits a covered line to the deepest test-directory frame, code reached through a helper-*.R/setup-*.R wrapper is attributed to the helper rather than the originating test-*.R file, and such mutants conservatively run the whole suite. "per_file" instruments the package once and runs the suite a single time through a reporter that snapshots coverage per test file, giving exact file-level attribution (no helper fallback) at roughly the same cost; it depends on covr internals, so it is opt-in.
target_margin: Optional desired half-width of the confidence interval on the mutation score, as a proportion (e.g. 0.05 for +/-5 percentage points). When set, the number of mutants to sample is derived from it using worst-case (p = 0.5) sizing at confidence, finite-population corrected and capped at the number of mutants generated (if the requested precision needs more mutants than exist, all are tested). Mutually exclusive with max_mutants. The required sample size depends on the target precision, not on program size (Gopinath et al., ISSRE 2015).
confidence: Confidence level for target_margin sizing and for the Wilson confidence interval reported on a sampled mutation score. Default 0.95.
max_show: Maximum number of surviving mutants to print to the console; the remainder are summarised as "... and N more" but always remain in the returned package_mutants. Use Inf to print every survivor. Default 50.

Value

An invisible list with four components:

package_mutants: Named list with mutant path, mutation info, status, and optional equivalence flags.
test_results: Named list mapping mutant IDs to statuses: "KILLED", "SURVIVED", or "HANG".
timing: Named list of phase durations in seconds: baseline, generation, test_execution, and equivalence_detection.
summary: Named list with generated, tested, killed, hanged, survived, mutation_score, mutation_score_ci (a length-2 percentage vector, or NULL when no sampling occurred), and confidence.

Details

The example is not run during routine automated checks because it creates and mutation-tests a throwaway package, which is too slow for that context.

Test strategy is, by default, detected automatically:

If tests/testthat/ exists, the mutant is loaded in-process with pkgload::load_all() (no installation) and its tests are run the way the package's own tests/testthat.R harness runs them, i.e. with the same arguments (notably any filter) that the harness passes to testthat::test_check(), via testthat::test_dir().
Otherwise, if inst/tinytest/ exists, the mutant is loaded in-process with pkgload::load_all() (no installation) and its tests are run with tinytest::run_test_dir("inst/tinytest").
Otherwise, if tests/ exists, mutator installs the mutant package with --install-tests and runs tools::testInstalledPackage().

Pass strategy to override this (for example to run a testthat or tinytest package through the slower installed-tests path for comparison).

Examples

# \donttest{
pkg <- file.path(tempdir(), "examplepkg")
dir.create(file.path(pkg, "R"), recursive = TRUE, showWarnings = FALSE)
dir.create(file.path(pkg, "tests", "testthat"), recursive = TRUE, showWarnings = FALSE)
writeLines(c(
  "Package: examplepkg",
  "Title: Example Package",
  "Version: 0.0.1",
  "Description: Minimal package for a mutator example.",
  "License: GPL-3",
  "Encoding: UTF-8"
), file.path(pkg, "DESCRIPTION"))
writeLines("export(add)", file.path(pkg, "NAMESPACE"))
writeLines("add <- function(x, y) x + y", file.path(pkg, "R", "add.R"))
writeLines(
  "testthat::expect_equal(add(1, 2), 3)",
  file.path(pkg, "tests", "testthat", "test-add.R")
)
result <- mutate_package(pkg, cores = 1, max_mutants = 1, timeout_seconds = 10)
#> Generated 1 AST-based mutants for add.R
#> Generated 1 mutants from 1 source files.
#> Running the test suites of 1 mutant...
#> 
#> Surviving mutants (1):
#>   R/add.R:1   '+' -> '-'
#>     > 1 | add <- function(x, y) x + y
#> Timing (seconds):
#>   Baseline run:          1.0
#>   Mutant generation:     0.0
#>   Test execution:        1.0
#>   Equivalence detection: 0.0
#> 
#> Mutation Testing Summary:
#>   Total mutants:    1
#>   Killed:           0
#>   Hanged:           0
#>   Survived:         1
#>   Mutation Score:   0.00%
names(result)
#> [1] "package_mutants" "test_results"    "timing"          "summary"        
# }