York University Viking Configuration
This config is for running nf-core pipelines, either directly on Viking or via tower.nf
To use, run the pipeline with -profile york_viking. This will download and launch the york_viking.config which has been pre-configured with a setup suitable for the Viking cluster.
I recommend running Nextflow via tower.nf, in which case make sure to include the york_viking profile within the pipeline.
However, should you need to run it directly on the HPC follow the guidance below.
Loading the required modules
Before running the pipeline you will need to load Nextflow and Singularity using the environment module system on the Viking cluster. You can do this by issuing the commands below:
## Load Nextflow and Singularity environment modules
module purge
module load Apptainer
module load Nextflow/23.10.0You may want to add those module load commands to your shell configuration file if you use them often.
Environmental variables
You need to make sure that the following environmental variables are set (pointing to appropriate dirs):
export NXF_HOME="/mnt/scratch/<PATH_TO_FOLDER>"
export NXF_CONDA_CACHEDIR="/mnt/scratch/<PATH_TO_FOLDER>"
export NXF_TEMP="/mnt/scratch/<PATH_TO_FOLDER>"
export NXF_APPTAINER_CACHEDIR="/mnt/scratch/<PATH_TO_FOLDER>"
export APPTAINER_TMPDIR="/mnt/scratch/<PATH_TO_FOLDER>"
export APPTAINER_CACHEDIR="/mnt/scratch/<PATH_TO_FOLDER>"Running the nextflow pipeline
Create a params.json file with the desired parameters for the pipeline. Place this file in the folder from which you wish to run the Nextflow pipeline. Then using a screen session:
screen -S run-nextflow
nextflow run nf-core/<pipeline_name> -profile york_viking -name <run_name> -params-file params.json -r <nf-core revision>Config file
/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Nextflow config file for York Viking Cluster for the SLURM login nodes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Author: Matthew Care
Mail: matthew.care@york.ac.uk
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*/
params {
config_profile_contact = "Matthew Care"
config_profile_description = "The University of York Viking profile "
config_profile_url = "https://vikingdocs.york.ac.uk/"
}
apptainer {
enabled = true
autoMounts = true
// the default is 20 minutes and fails with large images
pullTimeout = "3 hours"
}
process {
maxRetries = 3
clusterOptions = "--get-user-env --account=${System.getenv('USER_ACCOUNT')}"
// Get user environment and assign account
cache = "lenient"
afterScript = "sleep 60"
stageInMode = "symlink"
stageOutMode = "rsync"
scratch = 'false'
afterScript = "sleep 60"
}
executor {
name = "slurm"
queueSize = 200
submitRateLimit = "10/1sec"
exitReadTimeout = "30 min"
jobName = {
task.name
.replace(
"[",
"("
)
.replace(
"]",
")"
)
.replace(
" ",
"_"
)
}
}