.\" Automatically generated by Pandoc 2.9.2.1
.\"
.TH "fi_opx" "7" "2024\-03\-14" "Libfabric Programmer\[cq]s Manual" "Libfabric v1.21.0"
.hy
.PP
{%include JB/setup %}
.SH NAME
.PP
fi_opx - The Omni-Path Express Fabric Provider
.SH OVERVIEW
.PP
The \f[I]opx\f[R] provider is a native libfabric provider suitable for
use with Omni-Path fabrics.
OPX features great scalability and performance when running
libfabric-enabled message layers.
.PD 0
.P
.PD
OPX requires 3 additonal external development libraries to build:
libuuid, libnuma, and the Linux kernel headers.
.SH SUPPORTED FEATURES
.PP
The OPX provider supports most features defined for the libfabric API.
.PP
Key features include:
.TP
Endpoint types
The Omni-Path HFI hardware is connectionless and reliable.
The OPX provider only supports the \f[I]FI_EP_RDM\f[R] endpoint type.
.TP
Capabilities
Supported capabilities include \f[I]FI_MSG\f[R], \f[I]FI_RMA,
\f[R]FI_TAGGED\f[I], \f[R]FI_ATOMIC\f[I], \f[R]FI_NAMED_RX_CTX\f[I],
\f[R]FI_SOURCE\f[I], \f[R]FI_SEND\f[I], \f[R]FI_RECV\f[I],
\f[R]FI_MULTI_RECV\f[I], \f[R]FI_DIRECTED_RECV\f[I], \f[R]FI_SOURCE*.
.PP
Notes on \f[I]FI_DIRECTED_RECV\f[R] capability: The immediate data which
is sent within the \[lq]senddata\[rq] call to support
\f[I]FI_DIRECTED_RECV\f[R] for OPX must be exactly 4 bytes, which OPX
uses to completely identify the source address to an exascale-level
number of ranks for tag matching on the recv and can be managed within
the MU packet.
Therefore the domain attribute \[lq]cq_data_size\[rq] is set to 4 which
is the OFI standard minimum.
.TP
Modes
Two modes are defined: \f[I]FI_CONTEXT2\f[R] and \f[I]FI_ASYNC_IOV\f[R].
The OPX provider requires \f[I]FI_CONTEXT2\f[R].
.TP
Additional features
Supported additional features include \f[I]FABRIC_DIRECT\f[R],
\f[I]scalable endpoints\f[R], and \f[I]counters\f[R].
.TP
Progress
\f[I]FI_PROGRESS_MANUAL\f[R] and \f[I]FI_PROGRESS_AUTO\f[R] are
supported, for best performance, use \f[I]FI_PROGRESS_MANUAL\f[R] when
possible.
\f[I]FI_PROGRESS_AUTO\f[R] will spawn 1 thread per CQ.
.TP
Address vector
\f[I]FI_AV_MAP\f[R] and \f[I]FI_AV_TABLE\f[R] are both supported.
\f[I]FI_AV_MAP\f[R] is default.
.TP
Memory registration modes
Only \f[I]FI_MR_SCALABLE\f[R] is supported.
.SH UNSUPPORTED FEATURES
.TP
Endpoint types
Unsupported endpoint types include \f[I]FI_EP_DGRAM\f[R] and
\f[I]FI_EP_MSG\f[R].
.TP
Capabilities
The OPX provider does not support \f[I]FI_RMA_EVENT\f[R] and
\f[I]FI_TRIGGER\f[R] capabilities.
.SH LIMITATIONS
.PP
OPX supports the following MPI versions:
.PP
Intel MPI from Parallel Studio 2020, update 4.
Intel MPI from OneAPI 2021, update 3.
Open MPI 4.1.2a1 (Older version of Open MPI will not work).
MPICH 3.4.2 and later.
.PP
Usage:
.PP
If using with OpenMPI 4.1.x, disable UCX and openib transports.
OPX is not compatible with Open MPI 4.1.x PML/BTL.
.SH CONFIGURATION OPTIONS
.TP
\f[I]OPX_AV\f[R]
OPX supports the option of setting the AV mode to use in a build.
3 settings are supported: - table - map - runtime
.PP
Using table or map will only allow OPX to use FI_AV_TABLE or FI_AV_MAP.
Using runtime will allow OPX to use either AV mode depending on what the
application requests.
Specifying map or table however may lead to a slight performance
improvement depending on the application.
.PP
To change OPX_AV, add OPX_AV=table, OPX_AV=map, or OPX_AV=runtime to the
configure command.
For example, to create a new build with OPX_AV=table:
.PD 0
.P
.PD
OPX_AV=table ./configure
.PD 0
.P
.PD
make install
.PD 0
.P
.PD
.PD 0
.P
.PD
There is no way to change OPX_AV after it is set.
If OPX_AV is not set in the configure, the default value is runtime.
.SH RUNTIME PARAMETERS
.TP
\f[I]FI_OPX_UUID\f[R]
OPX requires a unique ID for each job.
In order for all processes in a job to communicate with each other, they
require to use the same UUID.
This variable can be set with FI_OPX_UUID=${RANDOM} The default UUID is
00112233445566778899aabbccddeeff.
.TP
\f[I]FI_OPX_FORCE_CPUAFFINITY\f[R]
Boolean (0/1, on/off, true/false, yes/no).
Causes the thread to bind itself to the cpu core it is running on.
Defaults to \[lq]No\[rq]
.TP
\f[I]FI_OPX_RELIABILITY_SERVICE_USEC_MAX\f[R]
Integer.
This setting controls how frequently the reliability/replay function
will issue PING requests to a remote connection.
Reducing this value may improve performance at the expense of increased
traffic on the OPX fabric.
Default setting is 500.
.TP
\f[I]FI_OPX_RELIABILITY_SERVICE_PRE_ACK_RATE\f[R]
Integer.
This setting controls how frequently a receiving rank will send ACKs for
packets it has received without being prompted through a PING request.
A non-zero value N tells the receiving rank to send an ACK for the last
N packets every Nth packet.
Used in conjunction with an increased value for
FI_OPX_RELIABILITY_SERVICE_USEC_MAX may improve performance.
.PP
Valid values are 0 (disabled) and powers of 2 in the range of 1-32,768,
inclusive.
.PP
Default setting is 64.
.TP
\f[I]FI_OPX_SELINUX\f[R]
Boolean (0/1, on/off, true/false, yes/no).
Set to true if you\[cq]re running a security-enhanced Linux.
This enables updating the Jkey used based on system settings.
Defaults to \[lq]No\[rq]
.TP
\f[I]FI_OPX_HFI_SELECT\f[R]
String.
Controls how OPX chooses which HFI to use when opening a context.
Has two forms: - \f[C]<hfi-unit>\f[R] Force OPX provider to use
\f[C]hfi-unit\f[R].
- \f[C]<selector1>[,<selector2>[,...,<selectorN>]]\f[R] Select HFI based
on first matching \f[C]selector\f[R]
.PP
Where \f[C]selector\f[R] is one of the following forms: -
\f[C]default\f[R] to use the default logic - \f[C]fixed:<hfi-unit>\f[R]
to fix to one \f[C]hfi-unit\f[R] -
\f[C]<selector-type>:<hfi-unit>:<selector-data>\f[R]
.PP
The above fields have the following meaning: - \f[C]selector-type\f[R]
The selector criteria the caller opening the context is evaluated
against.
- \f[C]hfi-unit\f[R] The HFI to use if the caller matches the selector.
- \f[C]selector-data\f[R] Data the caller must match (e.g.\ NUMA node
ID).
.PP
Where \f[C]selector-type\f[R] is one of the following: - \f[C]numa\f[R]
True when caller is local to the NUMA node ID given by
\f[C]selector-data\f[R].
- \f[C]core\f[R] True when caller is local to the CPU core given by
\f[C]selector-data\f[R].
.PP
And \f[C]selector-data\f[R] is one of the following: - \f[C]value\f[R]
The specific value to match - \f[C]<range-start>-<range-end>\f[R]
Matches with any value in that range
.PP
In the second form, when opening a context, OPX uses the
\f[C]hfi-unit\f[R] of the first-matching selector.
Selectors are evaluated left-to-right.
OPX will return an error if the caller does not match any selector.
.PP
In either form, it is an error if the specified or selected HFI is not
in the Active state.
In this case, OPX will return an error and execution will not continue.
.PP
With this option, it is possible to cause OPX to try to open more
contexts on an HFI than there are free contexts on that HFI.
In this case, one or more of the context-opening calls will fail and OPX
will return an error.
For the second form, as which HFI is selected depends on properties of
the caller, deterministic HFI selection requires deterministic caller
properties.
E.g.
for the \f[C]numa\f[R] selector, if the caller can migrate between NUMA
domains, then HFI selection will not be deterministic.
.PP
The logic used will always be the first valid in a selector list.
For example, \f[C]default\f[R] and \f[C]fixed\f[R] will match all
callers, so if either are in the beginning of a selector list, you will
only use \f[C]fixed\f[R] or \f[C]default\f[R] regardles of if there are
any more selectors.
.PP
Examples: - \f[C]FI_OPX_HFI_SELECT=0\f[R] all callers will open contexts
on HFI 0.
- \f[C]FI_OPX_HFI_SELECT=1\f[R] all callers will open contexts on HFI 1.
- \f[C]FI_OPX_HFI_SELECT=numa:0:0,numa:1:1,numa:0:2,numa:1:3\f[R]
callers local to NUMA nodes 0 and 2 will use HFI 0, callers local to
NUMA domains 1 and 3 will use HFI 1.
- \f[C]FI_OPX_HFI_SELECT=numa:0:0-3,default\f[R] callers local to NUMA
nodes 0 thru 3 (including 0 and 3) will use HFI 0, and all else will use
default selection logic.
- \f[C]FI_OPX_HFI_SELECT=core:1:0,fixed:0\f[R] callers local to CPU core
0 will use HFI 1, and all others will use HFI 0.
- \f[C]FI_OPX_HFI_SELECT=default,core:1:0\f[R] all callers will use
default HFI selection logic.
.TP
\f[I]FI_OPX_DELIVERY_COMPLETION_THRESHOLD\f[R]
Integer.
Will be deprecated.
Please use FI_OPX_SDMA_BOUNCE_BUF_THRESHOLD.
.TP
\f[I]FI_OPX_SDMA_BOUNCE_BUF_THRESHOLD\f[R]
Integer.
The maximum message length in bytes that will be copied to the SDMA
bounce buffer.
For messages larger than this threshold, the send will not be completed
until receiver has ACKed.
Value must be between 16385 and 2147483646.
Defaults to 16385.
.TP
\f[I]FI_OPX_SDMA_DISABLE\f[R]
Integer.
Disables SDMA offload hardware.
Default is 0
.TP
\f[I]FI_OPX_EXPECTED_RECEIVE_ENABLE\f[R]
Boolean (0/1, on/off, true/false, yes/no).
Enables expected receive rendezvous using Token ID (TID).
Defaults to \[lq]No\[rq].
This feature is not currently supported.
.TP
\f[I]FI_OPX_PROG_AFFINITY\f[R]
String.
This sets the affinity to be used for any progress threads.
Set as a colon-separated triplet as \f[C]start:end:stride\f[R], where
stride controls the interval between selected cores.
For example, \f[C]1:5:2\f[R] will have cores 1, 3, and 5 as valid cores
for progress threads.
Default is \f[C]1:4:1\f[R].
.TP
\f[I]FI_OPX_AUTO_PROGRESS_INTERVAL_USEC\f[R]
Integer, This setting controls the time (in usecs) between polls for
auto progress threads.
Default is 1.
.TP
\f[I]FI_OPX_PKEY\f[R]
Integer.
Partition key, a 2 byte positive integer.
Default is 0x8001
.TP
\f[I]FI_OPX_SL\f[R]
Integer.
Service Level.
This will also determine Service Class and Virtual Lane.
Default is 0
.SH SEE ALSO
.PP
\f[C]fabric\f[R](7), \f[C]fi_provider\f[R](7), \f[C]fi_getinfo\f[R](7),
.SH AUTHORS
OpenFabrics.