About Jülich Supercomputing Centre
Jülich Supercomputing Centre (JSC) provides supercomputer resources, IT tools, methods and know-how for the Jülich Research Centre, and for other European users through the John von Neumann Institute for Computing—a foundation that supports computational science in Germany and Europe. Jülich's JUGENE and JuRoPA/HPC-FF are currently the fastest computers in Europe, and occupy the third and tenth place in the world rankings respectively at top500.org.
To provide support for European scientific research involving large-scale computational simulation, JSC planned to create three new leading-edge supercomputers. JUGENE would be the fastest computer in Europe, based on proprietary IBM BlueGene/P technology.
JuRoPA and HPC-FF, two smaller clusters, would take a slightly different approach—using standard computer hardware (rack-mounted Bull NovaScale and Sun blade servers, based on the Intel Nehalem processor architecture) in a cluster. JuRoPA, the larger of the two clusters, has 17,664 processor cores distributed between 2,208 computing nodes, while HPC-FF, which is intended for use in atomic fusion research, has 8,640 cores and 1,080 nodes. The two supercomputers are based on similar technologies and can be combined to form a single cluster if required.
"Cluster architectures create their own unique challenges," said Ulrich Detert, Project Leader for JuRoPA. "Aside from hardware faults, the two major issues are latency and 'OS jitter'. When we were building the JuRoPA and HPC-FF supercomputers, we needed an operating system that would help us solve these problems."
The JuRoPA and HPC-FF supercomputers are among the most powerful HPC clusters ever built from commodity components. The commodity approach is not just limited to the hardware side—it also extends to the operating system. While many supercomputers run a specialised lightweight operating system on the compute nodes, both JuRoPA and HPC-FF rely on SUSE® Linux Enterprise Server as the operating system for all nodes.
"We chose SUSE Linux Enterprise Server as the basis for JuRoPA and HPC-FF because we saw it as a very stable and efficient platform," said Dr. Norbert Eicker, the designer of JuRoPA and HPC-FF. "Because it is open source, we were able to strip out all the features we did not need—for example, the graphical desktop manager and some of the kernel functions—which helped us to eliminate many of the unneeded system processes that cause OS jitter and increase latency."
SUSE Linux Enterprise Server also runs on the I/O and front-end nodes for JUGENE, while the main compute nodes run the ultra-lightweight IBM CNK operating system.
JSC is engaged in cooperative research with Novell using SUSE Linux Enterprise Real Time Extension to resolve the issue of OS jitter. This is a phenomenon caused by operating systems performing system operations (for example, daemon processes and interrupts) that create 'noise' or 'jitter', which interferes with the performance of parallel applications running in HPC clusters.
SUSE Linux Enterprise Real Time Extension is specifically designed to handle time-critical application workloads such as those found in high performance computing environments. By enabling applications to be prioritised over system processes, the software allows time-sensitive workloads to be processed reliably and predictably—keeping latency to a minimum.
"We see enormous potential in the field of real-time Linux," said Prof. Dr. Dr. Lippert, Head of Jülich Supercomputing Centre. "Novell is at the forefront of this technology, which holds significant benefits for high-performance computing environments."
The JuRoPA and HPC-FF supercomputers have now been fully built and tested, and are ready to take on project workload. HPC-FF will primarily be used in atomic fusion research, while JuRoPA will handle more than 200 different projects in a variety of areas of scientific research.
"Working with clusters is an excellent model for supercomputing because it enables a high level of scalability, and doesn't require expensive proprietary hardware," said Dr. Eicker. "It is a very effective architecture, especially if you can resolve the issues of OS jitter and high latency. Thanks to the customisability and real-time features of SUSE Linux Enterprise, we have been able to meet the challenge."
JSC is continuing its work on the new supercomputers—for example, improving hardware monitoring to mitigate the risk of hardware failure, and continuing to tune the SUSE Linux Enterprise Real Time Extension kernel to reduce the effects of OS jitter—and is confident that JuRoPA and HPC-FF will deliver the required performance and stability.
"Linux is the standard in high-performance computing," said Prof. Dr. Dr. Lippert. "By using SUSE Linux Enterprise, we are sure to get a reliable basis for our supercomputers and world-class support for sustained productivity."