Permutation-Based Testing
24 October 2023
Contents
By default, CaDrA performs both forward and backward search algorithm to look for a subset of features whose union is maximally associated with an outcome of interest, based on (currently) one of four scoring functions (Kolmogorov-Smirnov, Conditional Mutual Information, Wilcoxon, and custom-defined). To test whether the strength of the association between the set of features and the observed input scores (e.g., pathway activity, drug sensitivity, etc.) is greater than it would be expected by chance, CaDrA supports permutation-based significance testing. Importantly, the permutation test iterates over the entire search procedure (e.g., if top_N = 7, each permutation iteration will consist of running the search over the top 7 features).
1 Load packages
library(CaDrA)2 Load required datasets
- A
binary features matrixalso known asFeature Set(such as somatic mutations, copy number alterations, chromosomal translocations, etc.) The 1/0 row vectors indicate the presence/absence of ‘omics’ features in the samples. TheFeature Setcan be a matrix or an object of class SummarizedExperiment from SummarizedExperiment package) - A vector of continuous scores (or
Input Scores) representing a functional response of interest (such as protein expression, pathway activity, etc.)
# Load pre-simulated feature set
# See ?sim_FS for more information
data(sim_FS)
# Load pre-computed input-score
# See ?sim_Scores for more information
data(sim_Scores)3 Find a subset of features that maximally associated with a given outcome of interest
Here we are using Kolmogorow-Smirnow (KS) scoring method to search for best features
candidate_search_res <- CaDrA::candidate_search(
FS = sim_FS,
input_score = sim_Scores,
method = "ks_pval", # Use Kolmogorow-Smirnow scoring function
alternative = "less", # Use one-sided hypothesis testing
weights = NULL, # If weights is provided, perform a weighted-KS test
search_method = "both", # Apply both forward and backward search
top_N = 7, # Number of top features to kick start the search
max_size = 10, # Allow at most 10 features in meta-feature matrix
best_score_only = FALSE # Return meta-feature, its observed input scores and corresponding best score
)4 Visualize best meta-features result
# Extract the best meta-feature result
topn_best_meta <- CaDrA::topn_best(topn_list = candidate_search_res)
# Visualize meta-feature result
CaDrA::meta_plot(topn_best_list = topn_best_meta)
5 Perform permutation-based testing
# Set seed for permutation
set.seed(123)
perm_res <- CaDrA::CaDrA(
FS = sim_FS,
input_score = sim_Scores,
method = "ks_pval", # Use Kolmogorow-Smirnow scoring function
weights = NULL, # If weights is provided, perform a weighted-KS test
alternative = "less", # Use one-sided hypothesis testing
search_method = "both", # Apply both forward and backward search
top_N = 7, # Repeat the search with the top N features
max_size = 10, # Allow at most 10 features in the meta-feature matrix
n_perm = 1000, # Number of permutations to perform
plot = FALSE, # We will plot later
ncores = 4 # Number of cores to perform parallelization
)6 Visualize permutation result
# Visualize permutation results
permutation_plot(perm_res = perm_res)
7 SessionInfo
sessionInfo()
R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS
Matrix products: default
BLAS: /home/biocbuild/bbs-3.18-bioc/R/lib/libRblas.so
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB LC_COLLATE=C
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
time zone: America/New_York
tzcode source: system (glibc)
attached base packages:
[1] stats4 stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] CaDrA_1.0.0 testthat_3.2.0
[3] devtools_2.4.5 usethis_2.2.2
[5] SummarizedExperiment_1.32.0 Biobase_2.62.0
[7] GenomicRanges_1.54.0 GenomeInfoDb_1.38.0
[9] IRanges_2.36.0 S4Vectors_0.40.0
[11] BiocGenerics_0.48.0 MatrixGenerics_1.14.0
[13] matrixStats_1.0.0 BiocStyle_2.30.0
loaded via a namespace (and not attached):
[1] bitops_1.0-7 tcltk_4.3.1 remotes_2.4.2.1
[4] rlang_1.1.1 magrittr_2.0.3 compiler_4.3.1
[7] reshape2_1.4.4 callr_3.7.3 vctrs_0.6.4
[10] stringr_1.5.0 profvis_0.3.8 pkgconfig_2.0.3
[13] crayon_1.5.2 fastmap_1.1.1 magick_2.8.1
[16] XVector_0.42.0 ellipsis_0.3.2 labeling_0.4.3
[19] caTools_1.18.2 utf8_1.2.4 promises_1.2.1
[22] rmarkdown_2.25 sessioninfo_1.2.2 ps_1.7.5
[25] purrr_1.0.2 xfun_0.40 zlibbioc_1.48.0
[28] cachem_1.0.8 jsonlite_1.8.7 later_1.3.1
[31] DelayedArray_0.28.0 parallel_4.3.1 prettyunits_1.2.0
[34] R6_2.5.1 bslib_0.5.1 stringi_1.7.12
[37] pkgload_1.3.3 brio_1.1.3 jquerylib_0.1.4
[40] Rcpp_1.0.11 bookdown_0.36 iterators_1.0.14
[43] knitr_1.44 R.utils_2.12.2 R.cache_0.16.0
[46] httpuv_1.6.12 Matrix_1.6-1.1 tidyselect_1.2.0
[49] rstudioapi_0.15.0 abind_1.4-5 yaml_2.3.7
[52] doParallel_1.0.17 gplots_3.1.3 codetools_0.2-19
[55] miniUI_0.1.1.1 misc3d_0.9-1 processx_3.8.2
[58] pkgbuild_1.4.2 lattice_0.22-5 tibble_3.2.1
[61] plyr_1.8.9 shiny_1.7.5.1 withr_2.5.1
[64] evaluate_0.22 desc_1.4.2 urlchecker_1.0.1
[67] pillar_1.9.0 BiocManager_1.30.22 KernSmooth_2.23-22
[70] foreach_1.5.2 generics_0.1.3 rprojroot_2.0.3
[73] RCurl_1.98-1.12 ggplot2_3.4.4 munsell_0.5.0
[76] scales_1.2.1 gtools_3.9.4 xtable_1.8-4
[79] glue_1.6.2 ppcor_1.1 tools_4.3.1
[82] fs_1.6.3 grid_4.3.1 colorspace_2.1-0
[85] GenomeInfoDbData_1.2.11 cli_3.6.1 fansi_1.0.5
[88] S4Arrays_1.2.0 dplyr_1.1.3 gtable_0.3.4
[91] R.methodsS3_1.8.2 sass_0.4.7 digest_0.6.33
[94] SparseArray_1.2.0 farver_2.1.1 htmlwidgets_1.6.2
[97] R.oo_1.25.0 memoise_2.0.1 htmltools_0.5.6.1
[100] lifecycle_1.0.3 mime_0.12 MASS_7.3-60