HPC
Understanding Intel Omni-Path Architecture (Intel OPA)
November 8, 2016
3 min read
Intel OPA
With the growing needs of low latency, high bandwidth, and efficient performance in today’s HPC systems, Intel® Omni-Path Architecture introduces a new interconnect designed for HPC applications with scalability in mind. With 100Gb/s of bandwidth per port and port-to-port latencies similar to EDR InfiniBand, Omni-Path provides unique features that provide more robust traffic management and error detection than other RDMA fabrics
Unique features of Omni-Path
Omni-Path provides Packet Integrity Protection (PIP), which is a link-level error checking capability applied to all data going through the wire. It allows transparent detection and recovery of transmission errors as they occur. This aids in maintaining link continuity in the event of a lane failure by using Dynamic Lane Scaling (DLS) to use the remaining lanes in the link to continue operation. With Traffic Flow Optimization (TFO), the quality of service is improved by prioritizing data packets, regardless of packet ordering.
Cost Advantages
Reducing fabric costs was a major design goal for Omni-Path. One way to lower cost was to use denser switching with a 48-port switch chip. This equates to a 33 percent increase from the standard 36-port ASIC used for InfiniBand. With this density, less switches and cables are required in a system. In addition to savings in purchase expenses, power and maintenance costs are reduced as well.
Software Compatibility
Omni-Path uses the OpenFabrics Alliance interfaces that are used in software designed to run on InfiniBand. To these applications, Omni-Path is treated like any other RDMA fabric. Intel relies on True Scale software technology from QLogic for MPI. Existing MPI programs and libraries for True Scale that use the Performance Scaled Messaging (PSM) library work as-is without recompilation. Omni-Path also supports enhanced versions of the library (PSM2), so programs do not need to be updated to use these features. Because Omni-Path supports large fabrics, a recompilation of the application is needed to take advantage of the available larger scale. With PSM2 a superset of PSM, backward compatibility is kept. Intel provides Omni-Path Linux drivers and other software required for SUSE Linux Enterprise Server (SLES), Red Hat Enterprise Linux (RHEL), and other Linux operating systems. Intel also provides complete, open-source software solutions for switches and server nodes, including a system management suite, host software stack, a fabric management stack GUI, and a fabric management stack.
Host Integration
For server-side connectivity, Omni-path uses traditional host adapter cards. In the future, Xeon Phi processors and Xeon Processors will have an in-package host adapter configuration where the fabric ASIC is integrated into the processor socket. Eventually, the Omni-Path interface will be integrated directly into the processor. In doing-so, performance, power efficiency, and cost will be improved due to the removal of extra hardware required when a host adapter device is connected to a PCIe bus.
HPC
Understanding Intel Omni-Path Architecture (Intel OPA)
November 8, 20163 min read
Intel OPA
With the growing needs of low latency, high bandwidth, and efficient performance in today’s HPC systems, Intel® Omni-Path Architecture introduces a new interconnect designed for HPC applications with scalability in mind. With 100Gb/s of bandwidth per port and port-to-port latencies similar to EDR InfiniBand, Omni-Path provides unique features that provide more robust traffic management and error detection than other RDMA fabrics
Unique features of Omni-Path
Omni-Path provides Packet Integrity Protection (PIP), which is a link-level error checking capability applied to all data going through the wire. It allows transparent detection and recovery of transmission errors as they occur. This aids in maintaining link continuity in the event of a lane failure by using Dynamic Lane Scaling (DLS) to use the remaining lanes in the link to continue operation. With Traffic Flow Optimization (TFO), the quality of service is improved by prioritizing data packets, regardless of packet ordering.
Cost Advantages
Reducing fabric costs was a major design goal for Omni-Path. One way to lower cost was to use denser switching with a 48-port switch chip. This equates to a 33 percent increase from the standard 36-port ASIC used for InfiniBand. With this density, less switches and cables are required in a system. In addition to savings in purchase expenses, power and maintenance costs are reduced as well.
Software Compatibility
Omni-Path uses the OpenFabrics Alliance interfaces that are used in software designed to run on InfiniBand. To these applications, Omni-Path is treated like any other RDMA fabric. Intel relies on True Scale software technology from QLogic for MPI. Existing MPI programs and libraries for True Scale that use the Performance Scaled Messaging (PSM) library work as-is without recompilation. Omni-Path also supports enhanced versions of the library (PSM2), so programs do not need to be updated to use these features. Because Omni-Path supports large fabrics, a recompilation of the application is needed to take advantage of the available larger scale. With PSM2 a superset of PSM, backward compatibility is kept. Intel provides Omni-Path Linux drivers and other software required for SUSE Linux Enterprise Server (SLES), Red Hat Enterprise Linux (RHEL), and other Linux operating systems. Intel also provides complete, open-source software solutions for switches and server nodes, including a system management suite, host software stack, a fabric management stack GUI, and a fabric management stack.
Host Integration
For server-side connectivity, Omni-path uses traditional host adapter cards. In the future, Xeon Phi processors and Xeon Processors will have an in-package host adapter configuration where the fabric ASIC is integrated into the processor socket. Eventually, the Omni-Path interface will be integrated directly into the processor. In doing-so, performance, power efficiency, and cost will be improved due to the removal of extra hardware required when a host adapter device is connected to a PCIe bus.
