This chapter provides information about submitting and running groups of multiple operations with varying directives.
FlowGroup is a collection of one or more operation(s) with the
directives associated with that operation. An operation can be in multiple
groups, and the directives for an operation can be different in every group that
it is in. Practically, a
FlowGroup acts like a “meta-operation” that
can be submitted and run like an operation. In fact, within the source code,
each operation is wrapped by a singleton group that handles command generation
and resource requests for that operation.
FlowGroup is automatically created for each operation, users must
manually create groups that contain more than one operation. Below is an example that creates a
ex to contain operations
# project.py from flow import FlowProject class Project(FlowProject): pass ex = Project.make_group(name='ex') @ex @Project.operation def op1(job): pass @ex @Project.operation def op2(job): pass if __name__ == '__main__': Project().main()
A group is eligible if
at least one of its operations is eligible.
To execute or submit only
ex, use the option
select the group like you would for a regular operation.
To avoid wasting resources when submitting groups with multiple operations, make sure that you group operations that require similar resources or that cheaper operations do not run for long. Mixing GPU operations with highly parallel CPU ones will likely leave either the GPU or CPUs idle while the other type of operation is running.
Running groups in parallel¶
Groups can be run in parallel. During parallel execution of groups, the
combined resources (GPUs, MPI ranks, etc.) are requested. For example,
if a group contains two operations,
op2, and each
requests two GPUs, then four GPUs will be requested when run in parallel.
When running the same group in series, only two GPUs would be requested.
One of the features of
FlowGroup is the ability to assign custom directives
to an operation that activate in a given group context. This means that
groups can function as context-specific execution protocols for operations. To
configure group-specific operation directives, use the
@group.with_directives decorator provided by the result of
In the following example,
op1 requests one GPU if run by itself or two GPUs if run through the group
python project.py run -o ex).
# project.py from flow import FlowProject, directives class Project(FlowProject): pass ex = Project.make_group(name='ex') @ex.with_directives(directives=dict(ngpu=2)) @directives(ngpu=1) @Project.operation def op1(job): pass if __name__ == '__main__': Project().main()