For the complete documentation index, see llms.txt. This page is also available as Markdown.

DRAGEN Software Mode

Overview

DRAGEN Software Mode runs DRAGEN secondary analysis on general-purpose CPU servers. It does not require a DRAGEN FPGA card, so you can run DRAGEN on an existing high performance computing (HPC) cluster, on a commodity Linux server, or on a standard cloud instance.

Mapping and alignment, variant calling, and zip and unzip of FASTQ and BAM files run entirely in software. Processing time scales with thread count, which allows cost versus time tradeoffs.

Use Software Mode where FPGA hardware is not available and where turnaround time is not critical, or when you run pipelines such as Population Genotyping that are designed for non-FPGA compute. To compare Software Mode with FPGA Mode, and for all deployment choices, see Deployment Options.

DRAGEN Software Mode is introduced with DRAGEN v4.6. This page applies to v4.6 and later. Earlier DRAGEN versions do not offer Software Mode as a supported deployment.

Requirements

Requirement
Value

Operating system

Red Hat Enterprise Linux (RHEL) 8.x or 9.x, and RHEL-compatible Enterprise Linux distributions.

Architecture

x86-64

CPU instructions

AVX2 (Advanced Vector Extensions 2) is mandatory.

Threads

16 or more available threads are required. 64 is recommended.

Memory

120 GB or more available system memory is required. 256 GB is recommended.

Local storage

Fast local storage, ideally NVMe or SSD, on the compute node. Network-attached storage is supported but not recommended.

File handles

nofile limit of 72000 or more. See File handles and user processes.

User processes

nproc limit of 65535 or more. See File handles and user processes.

Network

Outbound HTTPS access to license.dragen.illumina.com at runtime.

Sizing Recommendations

DRAGEN is a natively multi-threaded application. The software automatically detects the number of system threads available, and aims to use the entire system's cores. The --num-threads option can be used to control how many processor cores DRAGEN will use (also referred to as vCPUs or Threads). DRAGEN can not be given RAM limits.

Memory recommendations for sizing your compute:

Use Case
RAM Recommended (*)

WGS

256GB

WGS high coverage (>200x)

384GB

WGS low pass (<15x)

128GB

Exome

128GB

T/N Exome

256GB

Panel

128GB

5-base WGS

256GB

5-base Panel

256GB

* Note: Processing time scales with number of cores.

  • 64 vCPU is recommended as good balance for most workflows, including Germline WGS.

  • Panels and exomes can be run on fewer vCPUs, with minimal run time impact (32 vCPU).

  • Large Somatic Tumor-Normal WGS samples can benefit from more cores, if turn around time is important (96-128 vCPU).

Use the latest available generation in your region. The following families are recommended for DRAGEN v4.6.

Cloud
Instance family

AWS

m8a, m8i

Azure

Dv7 series, Ev7 series

GCP

c4d, n4d

OCI

VM.Standard.E6.Flex

For provisioning, storage, and networking, refer to your cloud provider's documentation.

AVX2 support

Software Mode uses AVX2 vector instructions throughout the software mapper and the software variant calling engines. AVX2 has shipped in Intel processors since the Haswell architecture in 2013, and in AMD processors since 2015.

DRAGEN checks for AVX2 before any other startup work, so a failure produces a bare message on stderr with no DRAGEN log file:

To confirm support before you install, check the CPU flags:

Threads and memory

DRAGEN checks available threads and memory prior to running.

If the node has too few threads, DRAGEN reports the following, where <threads> is the value it detected:

If the node has too little memory, DRAGEN reports the following, where <memory> is the value it detected:

Both minimums are startup gates rather than hard hardware limits. --min-threads INT and --min-memory INT lower them, but Illumina does not recommend either option.

Network connectivity

Every Software Mode run authenticates and meters usage against the Illumina DRAGEN License server, so each compute node requires outbound internet connectivity at runtime. Analysis fails if the node cannot reach the license server.

Allow outbound HTTPS on port 443 to license.dragen.illumina.com from every node that runs DRAGEN, including all worker nodes in an HPC cluster.

Confirm connectivity from each node before you submit jobs:

Obtain and Install

Software Mode is available as a self-extracting installer named dragen-softwaremode-<version>.<rhel>.<arch>.bin and as an RPM. Download either format from the DRAGEN - Getting Started app, which you can open from the Illumina BioInsight Platform home page or at app.dragen.illumina.com. The steps below use the self-extracting installer.

1

Confirm the node meets the requirements

Verify the operating system, AVX2 support, recommended thread count, and available memory as described in Requirements.

2

Download the installer

In the DRAGEN - Getting Started app, download dragen-softwaremode-<version>.<rhel>.<arch>.bin for the RHEL major version and architecture of the target node. To fetch the installer directly on the node, copy the download link that the app provides.

3

Run the installer

By default the installer extracts to a directory beside itself:

The extracted target directory is $(pwd)/dragen-softwaremode-<version>.<rhel>.<arch>.

4

Configure licensing

Retrieve an Illumina BioInsight Platform API Key before your first run, and confirm the node can reach license.dragen.illumina.com. See Licensing and Network connectivity.

5

Confirm the installation

Run the DRAGEN binary from the extracted directory:

Enable Software Mode

There are two ways to put a run into Software Mode. Either one is sufficient, and both reach the same execution mode.

Dedicated package

The dragen-softwaremode installer enables Software Mode for every run on that installation, so no command line option is required. Build the command line exactly as you would for an FPGA Mode run:

The --sw-mode option

--sw-mode requests Software Mode explicitly for a single run when not using the dedicated Software Mode package.

Option
Type
Description

--sw-mode, -s

FLAG

Run DRAGEN in Software Mode.

--sw-mode takes no value. Its presence on the command line, or in a configuration file as sw-mode = true, is what activates the mode.

Licensing

Software Mode is licensed with an Illumina BioInsight Platform API Key. Usage is metered and charged in BioInsight Credits (BICs).

Because metering happens at runtime, each node needs network access to the license server. See Network connectivity.

For how to obtain a key and how to supply it to DRAGEN, see BioInsight Platform Licensing.

Pipelines not supported

DRAGEN Software Mode does not yet support the following:

Pipeline

TruPath

RNA

Spatial

FastQC

Performance and Tuning

Threads

In Software Mode, DRAGEN defaults to all cores available on the node, which differs from FPGA Mode. This default is also the recommended setting.

Option
Type
Description

--num-threads

INT

Number of processor threads to use. Must be a positive number. (default=all available cores in Software Mode)

Set --num-threads below the core count only when you deliberately share a node, or when you want to reserve cores for other work. DRAGEN enables thread affinity whenever --num-threads is set.

Local storage

DRAGEN analysis is I/O intensive, and disk throughput can limit performance, affecting total runtime.

  • Use fast local storage, ideally NVMe or SSD on the compute node, for inputs, outputs, and temporary files.

  • Keep the reference directory on local storage. Every node reads it on every run.

Network-attached storage is supported but not recommended.

Cloud storage

  • Input streaming of FASTQ files from S3 or Azure Blob Storage is supported.

  • Output streaming of temporary files and output files to S3 or Azure Blob Storage is not supported.

  • Cloud instances must use attached storage such as EBS or FSx for temporary and output files.

File handles and user processes

Software Mode opens more file handles than an FPGA Mode run, because more of the pipeline runs as host threads. Raise the limits before you run.

  • File handles: allow num_threads * (batch_size + reference + output file + 1 index file at a time), plus stdin, stdout, stderr, and the log file.

  • User processes: allow at least 65,000.

The values DRAGEN needs are above the default per-user maximum in Linux, so ulimit -n and ulimit -u cannot raise them on their own. Create a limits configuration file instead:

The raised limits apply to subsequent logins only. The current session does not get them, so log out and log back in before you run.

Confirm the limits in a new session:

Troubleshooting

ERROR: Your processor does not support Intel/AMD AVX2 vector instructions required by DRAGEN.

The CPU does not support AVX2. DRAGEN performs this check before it initializes logging, so there is no DRAGEN log file for the failed run and the message appears only on stderr.

Confirm CPU support with grep -o avx2 /proc/cpuinfo. If the command returns nothing, the node cannot run Software Mode. On a heterogeneous cluster, restrict submission to nodes with AVX2 using a scheduler constraint.

Available threads <threads> is less than the minimum number of threads required 16

The node exposes fewer than 16 threads, or the scheduler allocated fewer threads than the node has.

Run on a node with more threads available.

Available memory <memory>GB is less than the minimum system memory required 120GB

The node has less than 120 GB of system memory available.

Run on a node with more memory available.

ERROR: <pipeline> pipeline is not supported by Software Only mode.

The run requested a pipeline that is not available in Software Mode.

Run that analysis on DRAGEN FPGA Mode. See Pipelines not supported.

ERROR: Software mode (--sw-mode) can't be used with hardware ORA (--ora-use-hw=true)

The run requested hardware-accelerated ORA compression or decompression.

Omit --ora-use-hw, or set --ora-use-hw=false, to use the software ORA path.

ERROR: Software mode (--sw-mode) can't be used with hardware BCL (--bcl-use-hw=true, --bcl-conversion-only=true)

The run requested hardware BCL conversion. This also occurs when --bcl-conversion-only=true is set and --bcl-use-hw is left at a value that implies hardware.

Set --bcl-use-hw=false explicitly for BCL conversion in Software Mode.

An Illumina BioInsight Platform API Key must be provided.

No API key was found. DRAGEN looks for credentials in the license configuration file, the --api-key-file path, the DRAGEN_API_KEY_VALUE environment variable, and the DRAGEN user and internal default configuration files.

The full message lists both supported methods:

An API key file must contain a single line holding only the key. For setup instructions, see BioInsight Platform Licensing.

A run fails at licensing on some nodes but succeeds on others

The nodes that fail cannot reach the Illumina DRAGEN License server. This is a common symptom on HPC clusters where only login nodes have outbound internet access, or where a subset of worker nodes sits behind a different firewall rule.

Run the healthcheck from a failing node rather than from the login node:

Allow outbound HTTPS to license.dragen.illumina.com from every worker node. See Network connectivity.

For general DRAGEN troubleshooting, see Troubleshooting. For help from Illumina, see Support.

Last updated

Was this helpful?