Accelerating Large-scale Reservoir Simulations Using Supercomputers Zhangxin Chen Joint work with Hui Liu, Kun Wang, Jia Luo, Bo Yang and Ruijian He Slide 1 RESERVOIR SIMULATION GROUP Outline • Introduction • Parallel Reservoir Simulations • Case Studies • Conclusions Slide 2 RESERVOIR SIMULATION GROUP Introduction • Desktop/workstation • Multi-cores, OpenMP (CMG) • Efficient for small/medium cases • Long simulation time for large-scale cases with tens of millions of grid cells Slide 3 RESERVOIR SIMULATION GROUP Introduction • Clusters/supercomputers • OpenMP for shared memory - multi-cores, easy to use • MPI among computation nodes - communication - MPI-IO, read input and write output • Large-scale simulations Slide 4 RESERVOIR SIMULATION GROUP Reservoir Simulation • Reservoir simulators share common modules: - Grids - Data, including rock, fluid and other properties - Linear solvers and preconditioners - Well modeling and key words parsing - Input and output - Visualization - Memory and communication management Slide 5 RESERVOIR SIMULATION GROUP Research Goals • Develop a common platform for various simulators -- black oil, compositional and thermal models -- chemical reaction, combustion -- geomechnics, etc. • Simulate extremely large-scale cases with billions of grid cells • Scalable, thousands of processors • Investigate parallel linear solvers and preconditioners • Study parallel computing techniques Slide 6 RESERVOIR SIMULATION GROUP Parallel Reservoir Simulations • Structured hexahedral grids • Simple and easy to implement • Finite difference methods • Finite volume methods • Corner-point grids (optional) Slide 7 RESERVOIR SIMULATION GROUP Parallel Linear Solvers Model F(x) = 0 A * x = b Our platform provides • Solvers - Distributed matrices - GMRES(m), ORTHOMIN(m) - Distributed vectors - BiCGSTAB, CG - Matrix and vector management • Preconditioners - Parallel matrix-vector operations - RAS preconditioner - Global communication modules - ILU(k), ILUT and ILUC - AMG from Hypre - Constrained pressure residual (CPR) Slide 8 RESERVOIR SIMULATION GROUP CPR Preconditioner • Pressure block App, positive definite • Saturation block Ass • Well block Aww • Coupled matrices Slide 9 RESERVOIR SIMULATION GROUP CPR-FPF Preconditioner Slide 10 RESERVOIR SIMULATION GROUP Case Studies • IBM Blue Gene/Q Supercomputer • IBM US • PowerA2, 16 (+1) cores, 16 GB mem • 32,768 CPU cores Slide 11 RESERVOIR SIMULATION GROUP Case 1: SPE10, validation • Water-oil model • Grid: 60 x 220 x 85, 1.12 million • Highly heterogeneous permeability • 2000 days, max step: 100 days • One injection well, four producers • Bottom figure: permeability, Kx • Right figure: oil rate Slide 12 RESERVOIR SIMULATION GROUP Case 2: SPE10, three phases, validation • Standard black oil model using SPE10 geological model • Grid: 60 x 220 x 85, 1.12 million, 2000 days • Oil rate of well 1 and well 2 Slide 13 RESERVOIR SIMULATION GROUP Case 3: validation • Four components black oil model (API tracking model) • API tracking case from CMG (data file mxsmo019.dat) Slide 14 RESERVOIR SIMULATION GROUP Case 4: refined SPE10 • Oil-water model • Grid: 300 x 1100 x 425, 140 million • BiCGSTAB solver • CPR preconditioner • 10 days Slide 15 • Quasi-IMPES decoupling • Inexact Newton method, tol: 1e-2 • IBM Blue Gene/Q supercomputer MPI Steps Newton Solver Time (s) 256 27 102 119 21056.59 512 26 99 114 9321.31 1024 25 90 104 4235.68 2048 25 90 104 2110.16 4096 26 98 124 1265.42 RESERVOIR SIMULATION GROUP Case 4: refined SPE10, scalability Slide 16 RESERVOIR SIMULATION GROUP Case 5: refined SPE1 • Black oil model • Grid: 100 million • Inexact Newton method • tol: 1e-3 Slide 17 • Solver: BiCGSTAB • Preconditioner: CPR-FPF • ABF decoupling • 10 days RESERVOIR SIMULATION GROUP Case 5: refined SPE1, scalability Slide 18 RESERVOIR SIMULATION GROUP Case 6: Poisson • 6 billion grid cells • GMRES(30), 90 iterations • RAS preconditioner, overlap 1 • Grid generation, build linear system, assemble, and solve Slide 19 MPI Grid (s) Build (s) Assemble (s) Overall (s) 512 429.78 57.83 130.54 1829.18 1024 200.24 29.28 66.12 906.73 2048 106.83 14.83 34.03 463.59 4096 47.82 7.59 17.70 232.77 RESERVOIR SIMULATION GROUP Case 6: Poisson, scalability Slide 20 RESERVOIR SIMULATION GROUP Conclusions • A parallel platform has been developed -- reservoir simulations -- grids, data, linear solvers, preconditioners -- well modeling, key word parsing and visualization. • Our simulators can compute large-scale models. • The platform and our simulators have excellent scalability. • Parallel computing is powerful to reservoir simulations. Slide 21 RESERVOIR SIMULATION GROUP Sponsors Slide 22 RESERVOIR SIMULATION GROUP Thanks. Slide 23 RESERVOIR SIMULATION GROUP