DRAGEN Software Mode
Overview
DRAGEN Software Mode runs DRAGEN secondary analysis on general-purpose CPU servers. It does not require a DRAGEN FPGA card, so you can run DRAGEN on an existing high performance computing (HPC) cluster, on a commodity Linux server, or on a standard cloud instance.
Mapping and alignment, variant calling, and zip and unzip of FASTQ and BAM files run entirely in software. Processing time scales with thread count, which allows cost versus time tradeoffs.
Use Software Mode where FPGA hardware is not available and where turnaround time is not critical, or when you run pipelines such as Population Genotyping that are designed for non-FPGA compute. To compare Software Mode with FPGA Mode, and for all deployment choices, see Deployment Options.
Requirements
Operating system
Red Hat Enterprise Linux (RHEL) 8.x or 9.x, and RHEL-compatible Enterprise Linux distributions.
Architecture
x86-64
CPU instructions
AVX2 (Advanced Vector Extensions 2) is mandatory.
Threads
16 or more available threads are required. 64 is recommended.
Memory
120 GB or more available system memory is required. 256 GB is recommended.
Local storage
Fast local storage, ideally NVMe or SSD, on the compute node. Network-attached storage is supported but not recommended.
File handles
nofile limit of 72000 or more. See File handles and user processes.
User processes
nproc limit of 65535 or more. See File handles and user processes.
Licensing
Network
Outbound HTTPS access to license.dragen.illumina.com at runtime.
Sizing Recommendations
DRAGEN is a natively multi-threaded application. The software automatically detects the number of system threads available, and aims to use the entire system's cores. The --num-threads option can be used to control how many processor cores DRAGEN will use (also referred to as vCPUs or Threads). DRAGEN can not be given RAM limits.
Memory recommendations for sizing your compute:
WGS
256GB
WGS high coverage (>200x)
384GB
WGS low pass (<15x)
128GB
Exome
128GB
T/N Exome
256GB
Panel
128GB
5-base WGS
256GB
5-base Panel
256GB
* Note: Processing time scales with number of cores.
64 vCPU is recommended as good balance for most workflows, including Germline WGS.
Panels and exomes can be run on fewer vCPUs, with minimal run time impact (32 vCPU).
Large Somatic Tumor-Normal WGS samples can benefit from more cores, if turn around time is important (96-128 vCPU).
Recommended cloud instance types
Use the latest available generation in your region. The following families are recommended for DRAGEN v4.6.
AWS
m8a, m8i
Azure
Dv7 series, Ev7 series
GCP
c4d, n4d
OCI
VM.Standard.E6.Flex
For provisioning, storage, and networking, refer to your cloud provider's documentation.
AVX2 support
Software Mode uses AVX2 vector instructions throughout the software mapper and the software variant calling engines. AVX2 has shipped in Intel processors since the Haswell architecture in 2013, and in AMD processors since 2015.
DRAGEN checks for AVX2 before any other startup work, so a failure produces a bare message on stderr with no DRAGEN log file:
To confirm support before you install, check the CPU flags:
Threads and memory
DRAGEN checks available threads and memory prior to running.
If the node has too few threads, DRAGEN reports the following, where <threads> is the value it detected:
If the node has too little memory, DRAGEN reports the following, where <memory> is the value it detected:
Both minimums are startup gates rather than hard hardware limits. --min-threads INT and --min-memory INT lower them, but Illumina does not recommend either option.
Network connectivity
Every Software Mode run authenticates and meters usage against the Illumina DRAGEN License server, so each compute node requires outbound internet connectivity at runtime. Analysis fails if the node cannot reach the license server.
Allow outbound HTTPS on port 443 to license.dragen.illumina.com from every node that runs DRAGEN, including all worker nodes in an HPC cluster.
Confirm connectivity from each node before you submit jobs:
Obtain and Install
Software Mode is available as a self-extracting installer named dragen-softwaremode-<version>.<rhel>.<arch>.bin and as an RPM. Download either format from the DRAGEN - Getting Started app, which you can open from the Illumina BioInsight Platform home page or at app.dragen.illumina.com. The steps below use the self-extracting installer.
Confirm the node meets the requirements
Verify the operating system, AVX2 support, recommended thread count, and available memory as described in Requirements.
Configure licensing
Retrieve an Illumina BioInsight Platform API Key before your first run, and confirm the node can reach license.dragen.illumina.com. See Licensing and Network connectivity.
Enable Software Mode
There are two ways to put a run into Software Mode. Either one is sufficient, and both reach the same execution mode.
Dedicated package
The dragen-softwaremode installer enables Software Mode for every run on that installation, so no command line option is required. Build the command line exactly as you would for an FPGA Mode run:
The --sw-mode option
--sw-mode requests Software Mode explicitly for a single run when not using the dedicated Software Mode package.
--sw-mode, -s
FLAG
Run DRAGEN in Software Mode.
--sw-mode takes no value. Its presence on the command line, or in a configuration file as sw-mode = true, is what activates the mode.
Licensing
Software Mode is licensed with an Illumina BioInsight Platform API Key. Usage is metered and charged in BioInsight Credits (BICs).
Because metering happens at runtime, each node needs network access to the license server. See Network connectivity.
For how to obtain a key and how to supply it to DRAGEN, see BioInsight Platform Licensing.
Pipelines not supported
DRAGEN Software Mode does not yet support the following:
TruPath
RNA
Spatial
FastQC
Performance and Tuning
Threads
In Software Mode, DRAGEN defaults to all cores available on the node, which differs from FPGA Mode. This default is also the recommended setting.
--num-threads
INT
Number of processor threads to use. Must be a positive number. (default=all available cores in Software Mode)
Set --num-threads below the core count only when you deliberately share a node, or when you want to reserve cores for other work. DRAGEN enables thread affinity whenever --num-threads is set.
Local storage
DRAGEN analysis is I/O intensive, and disk throughput can limit performance, affecting total runtime.
Use fast local storage, ideally NVMe or SSD on the compute node, for inputs, outputs, and temporary files.
Keep the reference directory on local storage. Every node reads it on every run.
Network-attached storage is supported but not recommended.
Cloud storage
Input streaming of FASTQ files from S3 or Azure Blob Storage is supported.
Output streaming of temporary files and output files to S3 or Azure Blob Storage is not supported.
Cloud instances must use attached storage such as EBS or FSx for temporary and output files.
File handles and user processes
Software Mode opens more file handles than an FPGA Mode run, because more of the pipeline runs as host threads. Raise the limits before you run.
File handles: allow
num_threads * (batch_size + reference + output file + 1 index file at a time), plusstdin,stdout,stderr, and the log file.User processes: allow at least 65,000.
The values DRAGEN needs are above the default per-user maximum in Linux, so ulimit -n and ulimit -u cannot raise them on their own. Create a limits configuration file instead:
Confirm the limits in a new session:
Troubleshooting
For general DRAGEN troubleshooting, see Troubleshooting. For help from Illumina, see Support.
Last updated
Was this helpful?