Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

qq submit

The qq submit command is used to submit qq jobs to the batch system. It is qq’s equivalent of Infinity’s psubmit.

Quick comparison with psubmit

  • qq submit does not ask for confirmation; it behaves like psubmit (...) -y.

  • Options and parameters are specified differently. The only positional argument is the script name — everything else is an option. You can see all supported options using qq submit --help.

    Infinity:

    psubmit cpu run_script ncpus=8,walltime=12h,props=cl_zero -y

    qq:

    qq submit -q cpu run_script --ncpus 8 --walltime 12h --props cl_zero

  • Options can also be specified directly in the submitted script, or as a mix of in-script and command-line definitions. Command-line options always take precedence.

  • Unlike with psubmit, you do not have to execute qq submit directly from the directory with the submitted script. You can run qq submit from anywhere and provide the path to your script. The job’s input directory will always be the submitted script’s parent directory. Additionally, qq submit can submit multiple jobs at once.

  • qq submit has better support for multi-node jobs than psubmit as it allows specifying resource requirements per requested node.

Description

Submits one or more qq jobs to the batch system.

qq submit [OPTIONS] SCRIPT...

SCRIPT… — One or more paths to the scripts to submit.

The submitted script(s) must contain the qq run shebang. You can add it to your script by running qq shebang SCRIPT.

When the job is successfully submitted, qq submit creates a .qqinfo file for tracking the job’s state.

Options

General settings

-q, --queue TEXT — Name of the queue to submit the job to.

-s, --server TEXT — Name of the batch server to submit the job to. If not specified, the job is submitted to the default batch server. Only supported on Metacentrum-family clusters. Read more about specifying a server here.

--account TEXT — Account to use for the job. Required only in environments with accounting (e.g., IT4Innovations).

--job-type TEXT — Type of the job. Defaults to standard. Available types: ‘standard’, ‘loop’, ‘continuous’. Read more about job types here.

--exclude TEXT — Colon-, comma-, or space-separated list of files or directories that should not be copied to the working directory. Paths must be relative to the input directory.

--include TEXT — Colon-, comma-, or space-separated list of files or directories to copy into the working directory in addition to the input directory contents. These files are not copied back after job completion. Paths must be absolute or relative to the input directory. Ignored if the input directory is used as the working directory.

--depend TEXT — Comma- or space-separated list of job dependencies in the format ‘=<job_id>[:<job_id>…]’. Available types: after (after start), afterok (after success), afternotok (after failure/kill), afterany (after completion regardless of outcome). Multiple job IDs in one expression (colon-separated) require all listed jobs to satisfy the condition. Multiple expressions must all be satisfied before the job starts. Examples: ‘afterok=1234’, ‘after=456:789’, ‘afterok=123,afternotok=678’. Read more about dependencies here.

--transfer-mode TEXT — Colon-, comma-, or space-separated list of transfer modes controlling when working directory files are transferred to the input directory. Modes: success (exit code 0), failure (non-zero exit code), always, never, or a specific exit code number (e.g., 42). Combine modes; files transfer if any apply. Defaults to success. On transfer, the working directory is deleted; otherwise it is preserved. Killed jobs are never transferred automatically. Ignored if the input directory is used as the working directory. Examples: ‘success’, ‘always’, ‘success:42’, ‘1 2 3’. Read more about transfer modes here.

--interpreter TEXT — Executable name or absolute path of the interpreter used to run the submitted script, including options for the interpreter. The interpreter must be available on the computing node. Defaults to bash. Read more about specifying interpreters here.

--batch-system TEXT — Name of the batch system used to submit the job. If not specified, the value of the environment variable ‘QQ_BATCH_SYSTEM’ is used or the system is auto-detected.

Requested resources

Memory and storage sizes are specified as ‘N’ where unit is one of b, kb, mb, gb, tb, pb (e.g., 500mb, 32gb).

Job resources are described in more detail in the Job resources section.

--nnodes INTEGER — Number of nodes to allocate for the job.

--ncpus-per-node INTEGER — Number of CPU cores to allocate per node.

--ncpus INTEGER — Total number of CPU cores to allocate for the job. Overrides –ncpus-per-node.

--mem-per-cpu TEXT — Memory to allocate per CPU core.

--mem-per-node TEXT — Memory to allocate per node. Overrides –mem-per-cpu.

--mem TEXT — Total memory to allocate for the job. Overrides –mem-per-cpu and –mem-per-node.

--ngpus-per-node INTEGER — Number of GPUs to allocate per node.

--ngpus INTEGER — Total number of GPUs to allocate for the job. Overrides –ngpus-per-node.

--walltime TEXT — Maximum runtime for the job. Examples: ‘1d’, ‘12h’, ‘10m’, ‘24:00:00’.

--work-dir, --workdir TEXT — Type of working directory to use for the job. Available types depend on the environment.

--work-size-per-cpu, --worksize-per-cpu TEXT — Storage to allocate per CPU core.

--work-size-per-node, --worksize-per-node TEXT — Storage to allocate per node. Overrides –work-size-per-cpu.

--work-size, --worksize TEXT — Total storage to allocate for the job. Overrides –work-size-per-cpu and –work-size-per-node.

--props TEXT — Colon-, comma-, or space-separated list of node properties required (e.g., cl_two) or prohibited (e.g., ^cl_two) to run the job.

Settings for continuous and loop jobs

Only used when job-type is continuous or loop.

--resubmit-from TEXT — Colon-, comma-, or space-separated ordered list of hosts to try resubmitting from. The job is resubmitted from the first reachable host. Allowed values: input (the submission machine), working (the execution node), or a specific hostname (e.g., perian.metacentrum.cz). Default value depends on the batch system. Examples: ‘input’, ‘input,working’, ‘input:st1:st2’, ‘working perian.metacentrum.cz’. Read more about resubmission hosts here.

Settings for loop jobs

Only used when job-type is loop.

--loop-start INTEGER — Starting cycle for a loop job. Defaults to 1.

--loop-end INTEGER — Ending cycle for a loop job.

--archive TEXT — Directory name for archiving files from a loop job. Defaults to storage.

--archive-format TEXT — Filename format for archived files. Defaults to job%04d.

--archive-mode TEXT — Colon-, comma-, or space-separated list of archive modes controlling when working directory files are archived upon job completion. Supports the same modes as –transfer-mode. Defaults to success.

Specifying options in the script

Instead of specifying submission options on the command line, you can include them directly in the script using qq directives.

qq directives follow this format: # qq <option>=<value> or # qq <option> <value> (both are equivalent).

The word qq is case-insensitive (qq, QQ, Qq, and qQ are all valid), and spacing is flexible. All qq directives must appear at the beginning of the script, before any executable commands.

Example:

#!/usr/bin/env -S qq run

# qq queue gpu
# qq job-type loop
# qq loop-end 10
# qq archive storage
# qq archive-format md%04d

# qq ncpus 8
# qq ngpus 1
# qq walltime 1d

metamodule add ...

Note

In the example above, kebab-case is used for option names, but qq directives also support snake_case, camelCase, and PascalCase. For example: # qq job-type loop, # qq job_type loop, # qq jobType loop, and # qq JobType loop are all equivalent.

All options of qq submit can be defined within the script body. Options that have a short form, such as -q/--queue and -s/--server, must be written in their long form (e.g., # qq queue gpu instead of # qq q gpu).

Command-line options always take precedence over options defined in the script body.

Submitting multiple jobs

You can submit multiple jobs by providing multiple script paths to qq submit. The command line options will apply to all submitted scripts, but each script can also define its own qq directives. Note however that command line options override options defined in the script body.

Submission of multiple jobs is parallelized and consequently much faster than submitting them one by one. Note however, that the order of the jobs is not guaranteed to be preserved, i.e. a script which you provide later in the list of scripts may be submitted before a script which you provided earlier.

Examples

qq submit run_script.sh -q cpu --ncpus 8 --workdir scratch_local --worksize-per-cpu 2gb --walltime 2d --props hyperthreading

Submits the script run_script.sh to the cpu queue, requesting 8 CPU cores and 16 GB of local scratch space (2 GB per core). The requested walltime is 48 hours, and the job must run on a node with the hyperthreading property. Additional options may come from the script or queue defaults, but command-line options take precedence.

qq submit run_script.sh

Submits the script run_script.sh, taking all submission options from the script itself or from queue/server defaults.

qq submit job*/run_script.sh -q cpu --ncpus 8 --walltime 1d

Submits all run_script.sh scripts in the job* directories (e.g., job1/run_script.sh, job2/run_script.sh) to the cpu queue, requesting 8 CPU cores and a walltime of 1 day for each of them. Additional parameters may be specified inside the individual scripts. The submitted jobs are completely independent of each other.