Skip to contents

R-CMD-check Codecov test coverage

mutator

Overview

mutator is an automated mutation testing tool for the R language. It applies mutation testing principles to help developers improve test suite quality by introducing small, systematic changes (mutations) to source code and verifying if tests can detect these changes.

Features

  • Comprehensive Mutation Testing: Applies various mutation operators to R source code
  • AST-Based Mutation: Uses Abstract Syntax Tree analysis for intelligent code mutations
  • Parallel Test Execution: Runs tests in parallel for improved performance
  • Equivalent Mutant Detection: Uses AI to identify mutants that are functionally equivalent to the original code
  • Detailed Reporting: Provides mutation scores and analysis of test suite effectiveness

Package Structure 

mutator/
├── R/                      # R source code
│   ├── mutator.R           # Core functionality
│   └── init.R
├── src/                    # C++ source + Catch2 C++ tests (testthat integration)
├── tests/                  # R tests (includes compiled C++ test entrypoint)
└── DESCRIPTION             # Package metadata

Installation 

# Install dependencies
install.packages(c("pkgload", "testthat", "httr", "jsonlite", "future", "furrr"))

# Install from GitHub (if available)
devtools::install_github("PRL-PRG/mutator")

# Or install from local source
# In R console
setwd("path/to/mutator")
devtools::install()

Quick Start Guide 

library(mutator)

# Mutate a single file
mutants <- mutate_file("path/to/your/file.R")

# Optional: cap returned mutants by random selection
mutants <- mutate_file("path/to/your/file.R", max_mutants = 20)

# Mutate an entire package and run tests
result <- mutate_package("path/to/your/package")

# Optional: cap tested mutants across the whole package
result <- mutate_package("path/to/your/package", max_mutants = 100)

# Optional: set a fixed timeout (seconds) per mutant test run
result <- mutate_package("path/to/your/package", timeout_seconds = 60)

# Optional: control where mutant files are written
result <- mutate_package("path/to/your/package", mutation_dir = tempdir())

Testing

mutator selects a package test strategy automatically:

  • If tests/testthat/ exists, mutator uses testthat::test_dir("tests/testthat").
  • Otherwise, if tests/ exists, mutator falls back to tools::testInstalledPackage(..., types = "tests") after installing each mutant with --install-tests.

The fallback path supports non-testthat layouts (for example tinytest-driven packages that run through tests/ scripts), but it is slower because each mutant must be installed before tests are executed.

Each mutant test run uses a timeout. By default, mutator runs the baseline suite first and derives the per-mutant timeout as baseline_elapsed_seconds * 1.5. You can override this by setting timeout_seconds explicitly.

Mutant outcomes are reported as:

  • SURVIVED: tests passed for the mutant
  • KILLED: tests failed (or execution error)
  • HANG: mutant exceeded timeout

mutator itself uses testthat for its own R tests and testthat + Catch2 for C++ tests.

  • C++ tests are located in src/test-*.cpp
  • The C++ test runner is src/test-runner.cpp
  • C++ tests are executed from tests/testthat/test-cpp.R via run_cpp_tests("mutator")

Run the full test suite with:

devtools::test()

Configuration

Timeouts and the contended baseline

Each mutant is run with a wall-clock timeout; exceeding it is reported as HANG. The timeout is derived from how long the package’s own test suite takes, unless you pass timeout_seconds explicitly.

Because mutants run in parallel (cores at a time), the timeout must account for contention: with many workers, each test run is slower than it would be alone — for packages that load many dependencies, or recompile C code on every R CMD INSTALL, dramatically so. A timeout based on a single, uncontended baseline run would then fire on nearly every mutant (a wave of false HANGs).

To avoid this, mutate_package() first runs the baseline suite once on its own (to confirm it passes and fail fast otherwise), then runs it cores times concurrently and takes the slowest of those as the contended baseline. The timeout is max(contended_baseline * 1.5, 5s). This self-calibrates to the machine, the chosen parallelism, and the package’s real load/compile cost — no manual tuning. Pass timeout_seconds to override it entirely. (When cores = 1, or when forking is unavailable, the solo baseline is used.)

CRAN mode (test selection)

By default mutate_package() runs each mutant’s tests in CRAN mode (cran = TRUE): the NOT_CRAN environment variable is set to "false" in the test subprocess, so testthat::skip_on_cran() and skip_if_offline() guards take effect and mutator runs the same tests CRAN would. This skips the slow, flaky, or network-dependent tests that packages mark as CRAN-skippable — which keeps mutation runs fast and avoids spurious timeouts/kills from, e.g., tests that hit the network.

Set cran = FALSE to run the full suite instead (NOT_CRAN = "true", the behaviour of devtools::test()):

mutate_package("path/to/pkg", cran = FALSE)

This applies to both test strategies (the testthat strategy and the installed-tests fallback). Note that it only affects tests the package explicitly guards with skip_on_cran() / skip_if_offline(); a package whose network test has no such guard will still run that test in either mode.

Equivalent Mutant Detection

Equivalent-mutant detection calls an OpenAI-compatible Chat Completions API. Configure it in any of these ways (listed highest precedence first; each setting is resolved independently, so they can be mixed):

  1. Programmatically, in your R session:

    set_openai_config(
  api_key  = "sk-...",
  model    = "gpt-4",
  base_url = "https://api.openai.com/v1" # any OpenAI-compatible endpoint
)

  2. A .openai_config file in the working directory (or in a directory you pass via get_openai_config(dir = ...)). It is a plain, human-readable file of field: value lines and is parsed, never executed:

    api_key: sk-...
model: gpt-4
base_url: https://api.openai.com/v1

    Only the given directory is consulted — parent directories are not searched.

  3. Environment variables OPENAI_API_KEY, OPENAI_MODEL and OPENAI_BASE_URL.

If nothing is configured, the model defaults to gpt-4 and the base URL to the public OpenAI API. Set base_url to target a self-hosted or alternative OpenAI-compatible service (for example http://localhost:11434/v1).

Survived mutants are analyzed in bounded batches (default 25 per request), and each mutant is shown to the model as a small unified diff of its edit (plus a short change label) rather than its full mutated source — compact, unambiguous, and in a format LLMs read natively. When mutate_package() runs with cores > 1 the batches are sent concurrently, which (with the bounded size) keeps the equivalence pass fast and avoids the truncated responses that otherwise drop verdicts.

For mutants the model flags as EQUIVALENT (the rare, high-stakes calls — they are excluded from the adjusted mutation score), it also returns a one-sentence reason, stored as equivalence_reason on the mutant so the call can be audited. No reason is requested for NOT_EQUIVALENT/DONT_KNOW, keeping responses small.

Progress and the results summary are emitted via message() (so they can be silenced with suppressMessages()). The full prompts and model responses are not printed by default; set options(mutator.verbose = TRUE) to log them.

Mutation Operators

mutator implements a wide range of mutation operators to thoroughly test your code:

Arithmetic and Logical Operator Replacements

Operator Description Example
cxx_add_to_sub Replaces + with - a + b → a - b
cxx_sub_to_add Replaces - with + a - b → a + b
cxx_mul_to_div Replaces * with / a * b → a / b
cxx_div_to_mul Replaces / with * a / b → a * b
cxx_eq_to_ne Replaces == with != a == b → a != b
cxx_ne_to_eq Replaces != with == a != b → a == b
cxx_gt_to_ge Replaces > with >= a > b → a >= b
cxx_gt_to_le Replaces > with <= a > b → a <= b
cxx_lt_to_le Replaces < with <= a < b → a <= b
cxx_lt_to_ge Replaces < with >= a < b → a >= b
cxx_ge_to_gt Replaces >= with > a >= b → a > b
cxx_ge_to_lt Replaces >= with < a >= b → a < b
cxx_le_to_lt Replaces <= with < a <= b → a < b
cxx_le_to_gt Replaces <= with > a <= b → a > b
cxx_and_to_or Replaces & with \| a & b → a \| b
cxx_or_to_and Replaces \| with & a \| b → a & b
cxx_logical_and_to_or Replaces && with \|\| a && b → a \|\| b
cxx_logical_or_to_and Replaces \|\| with && a \|\| b → a && b

Unary Operator Mutations

Operator Description Example
cxx_minus_to_noop Removes unary minus -x → x
cxx_remove_negation Removes logical negation !x → x

Assignment and Value Mutations

Operator Description Example
cxx_assign_const Replaces assignment with constant a = b → a = 42
cxx_replace_scalar_call Replaces function call with constant f(x) → 42
scalar_value_mutator Replaces constants 0 → 42, non-zero → 0
negate_mutator Negates conditionals x → !x, !x → x

Dependencies

mutator depends on:

  • R Packages:
    • pkgload: For loading mutated package copies
    • testthat: For test execution
    • xml2: For running C++ tests through testthat::run_cpp_tests()
    • future and furrr: For parallel execution
    • httr and jsonlite: For OpenAI API integration
  • LinkingTo: testthat (for Catch2 C++ test headers)
  • C++17: For native mutation engine implementation