HPC
How Ceph Software Storage Can Support the Explosion of AI/ML Data
October 20, 2021
5 min read
What is Ceph?
Ceph is open source software defined storage (SDS) designed to provide highly scalable object-, block- and file-based storage under a unified system, setting it apart from other SDS solutions. It allows decoupling data from physical hardware storage, using software abstraction layers, providing scaling and fault management capabilities.
As a distributed storage framework, Ceph has typically been used for high bandwidth, medium latency types of applications, such as content delivery, archive storage, or block storage for virtualization.
Its inherent scale-out support allows an organization to build large systems as demand increases. Additionally, it supports enterprise-grade features such as erasure coding, thin provisioning, cloning, load-balancing, automated tiering between flash and hard drives, and simplified maintenance and debugging.
Ceph for HPC
Even with its accessibility and flexibility, Ceph was not normally considered for HPC due to perceived poor performance in such an environment and its difficulty of use. However, a large community of open source Ceph developers is continuously adding improved management features. The Ceph Foundation provides an open, collaborative, and neutral home for stakeholders to coordinate their development and community investments in the Ceph ecosystem.
Now the world-leader in task-specific data center solutions, SoftIron®, is focused on designing and creating enterprise storage solutions with SDS-dedicated Ceph appliances that optimize Ceph and make it easier to use and adopt within the enterprise. So, it was natural to partner with Exxact to create solutions for the modern-day enterprise that make software-defined storage (SDS) simple.
Interested in a SoftIron Storage Solution?
Learn more about Ceph-based Storage Appliances on Exxact
SoftIron HyperDrive®
SoftIron’s HyperDrive Storage Node is a custom designed, dedicated Ceph appliance, custom built for SDS. All components integrate with and exploit Ceph’s outstanding functionality, resulting in a scale-out storage solution that runs at wire-speed at less than 100W per 1U node. It also outperforms traditional Xeon hardware in read/write performance. See the results.
To address complexity issues, HyperDrive Storage Manager was designed as a powerful, unified, and intuitive system to simplify the management of Ceph software and storage hardware radically.
With SoftIron HyperDrive, Exxact can design and build appliance based solutions from the ground up to ensure every component works at its peak, seamlessly and optimally with everything else in a wire speed chain.
In addition, HyperDrive Storage Router removes Ceph’s inability to support other files shares and protocols, and allows everyone to leverage the power of Ceph, no matter who the end user is.
Storage Router is a highly intelligent services gateway that enables enterprise-wide adoption of Ceph by consolidating user shares, virtualization, and any other storage technologies onto one scalable, high-performance platform.
To compute erasure coding on the fly, there is the recent addition of a hardware acceleration I/O module that uses programmable silicon. Dubbed “Accepherator,” the module solves the traditional “3X replication” data redundancy issue without the expense of high-end CPUs, while also sitting inline as a 10GbE network interface.
Creating three copies of data and spreading it across multiple drives can obviously get costly since 3X the usable storage is needed. That is where erasure coding steps in and processes data through an algorithm that breaks the data up into chunks and writes a single copy with extra parity bits that can be used to rebuild data should any media be lost.
Final Thoughts
Ceph can be your secret weapon to fight the ongoing battle to manage, grow and optimize your data center.
SDS solutions that use Ceph for HPC offer a way to build scalable, purpose-built, low-powered, wire-speed storage platforms that meet performance and costs requirements for increased use of artificial intelligence (AI), machine learning (ML), and analytics on large datasets, as well as increasingly sophisticated/realistic simulations and modeling.
Have any questions about the benefits of Ceph storage for HPC?
Contact Exxact Today
HPC
How Ceph Software Storage Can Support the Explosion of AI/ML Data
October 20, 20215 min read
What is Ceph?
Ceph is open source software defined storage (SDS) designed to provide highly scalable object-, block- and file-based storage under a unified system, setting it apart from other SDS solutions. It allows decoupling data from physical hardware storage, using software abstraction layers, providing scaling and fault management capabilities.
As a distributed storage framework, Ceph has typically been used for high bandwidth, medium latency types of applications, such as content delivery, archive storage, or block storage for virtualization.
Its inherent scale-out support allows an organization to build large systems as demand increases. Additionally, it supports enterprise-grade features such as erasure coding, thin provisioning, cloning, load-balancing, automated tiering between flash and hard drives, and simplified maintenance and debugging.
Ceph for HPC
Even with its accessibility and flexibility, Ceph was not normally considered for HPC due to perceived poor performance in such an environment and its difficulty of use. However, a large community of open source Ceph developers is continuously adding improved management features. The Ceph Foundation provides an open, collaborative, and neutral home for stakeholders to coordinate their development and community investments in the Ceph ecosystem.
Now the world-leader in task-specific data center solutions, SoftIron®, is focused on designing and creating enterprise storage solutions with SDS-dedicated Ceph appliances that optimize Ceph and make it easier to use and adopt within the enterprise. So, it was natural to partner with Exxact to create solutions for the modern-day enterprise that make software-defined storage (SDS) simple.
Interested in a SoftIron Storage Solution?
Learn more about Ceph-based Storage Appliances on Exxact
SoftIron HyperDrive®
SoftIron’s HyperDrive Storage Node is a custom designed, dedicated Ceph appliance, custom built for SDS. All components integrate with and exploit Ceph’s outstanding functionality, resulting in a scale-out storage solution that runs at wire-speed at less than 100W per 1U node. It also outperforms traditional Xeon hardware in read/write performance. See the results.
To address complexity issues, HyperDrive Storage Manager was designed as a powerful, unified, and intuitive system to simplify the management of Ceph software and storage hardware radically.
With SoftIron HyperDrive, Exxact can design and build appliance based solutions from the ground up to ensure every component works at its peak, seamlessly and optimally with everything else in a wire speed chain.
In addition, HyperDrive Storage Router removes Ceph’s inability to support other files shares and protocols, and allows everyone to leverage the power of Ceph, no matter who the end user is.
Storage Router is a highly intelligent services gateway that enables enterprise-wide adoption of Ceph by consolidating user shares, virtualization, and any other storage technologies onto one scalable, high-performance platform.
To compute erasure coding on the fly, there is the recent addition of a hardware acceleration I/O module that uses programmable silicon. Dubbed “Accepherator,” the module solves the traditional “3X replication” data redundancy issue without the expense of high-end CPUs, while also sitting inline as a 10GbE network interface.
Creating three copies of data and spreading it across multiple drives can obviously get costly since 3X the usable storage is needed. That is where erasure coding steps in and processes data through an algorithm that breaks the data up into chunks and writes a single copy with extra parity bits that can be used to rebuild data should any media be lost.
Final Thoughts
Ceph can be your secret weapon to fight the ongoing battle to manage, grow and optimize your data center.
SDS solutions that use Ceph for HPC offer a way to build scalable, purpose-built, low-powered, wire-speed storage platforms that meet performance and costs requirements for increased use of artificial intelligence (AI), machine learning (ML), and analytics on large datasets, as well as increasingly sophisticated/realistic simulations and modeling.
Have any questions about the benefits of Ceph storage for HPC?
Contact Exxact Today
