Imagine the challenges involved in solving a problem with 600 billion nonlinear equations. Only the largest supercomputers can handle that complexity. Moreover, traditional solvers cannot be used: they do not achieve the high resolution and scalability needed.

These were the challenges tackled by a team of scientists at the University of Texas at Austin and IBM, in collaboration with New York University and Cal Tech. Their goal was to simulate the Earth’s mantle. And their extraordinary achievement was recently recognized by their winning the prestigious Gordon Bell Prize.

The research involved creating a novel hybrid solver that exhibits optimal algorithmic performance and achieves extreme scaling for difficult nonlinear partial differential equations. Rather than writing all the layers of complex mathematical software needed from scratch, the research team utilized Argonne National Laboratory’s Portable, Extensible Toolkit for Scientific Computation (PETSc) for its tried-and-true implementation of standard algorithms needed within the newly developed hybrid solver.

PETSc is a suite of data structures and routines developed at Argonne for the scalable solution of applications modeled by partial differential equations. PETSc includes parallel Krylov and algebraic multigrid solvers, both of which were used in the winning Gordon Bell Prize application.

“Krylov methods are often used to accelerate the convergence of iterative schemes for sparse linear operations, but they involve well-known scaling difficulties,” said Barry Smith, a senior computational mathematician in the Mathematics and Computer Science Division at Argonne and developer of PETSc. ​“By using PETSc’s algebraic multigrid on only the coarsest levels of the hierarchy, the researchers were able to retain superior convergence without having to write a lot of new code.”

Other innovations made by the team included redesigning various algorithms and solvers and adding mixed continuous-discontinuous discretizations and a hybrid spectral-geometric multigrid method. The resulting high-resolution mantle flow model proved remarkably successful. Not only was it able to handle highly adapted meshes, extreme degrees of nonlinearity and ill-conditioning, and a wide range of material properties, but it scaled to 1.5 million cores on a leadership-class computer – a 96-fold increase in problem size – while maintaining 97% parallel efficiency.

The work enabled scientists for the first time to simulate the most extreme tomography features of the Earth’s surface.

This was also the first time that a Gordon Bell Prize winner used PETSc, according to Smith. ​“Over the past decade, several applications that used PETSc have won Gordon Bell awards in special categories, but not the Gordon Bell Prize in High Performance Computing. We are delighted to have had this small part in this achievement.”

The Gordon Bell Prize in High Performance Computing is awarded annually in recognition of an outstanding achievement in high-performance computing that helps solve critical science and engineering problems. This year’s prize was presented at the SC15 international supercomputing conference.