Accelerating Large-scale Reservoir Simulations
Using Supercomputers
Zhangxin Chen
Joint work with Hui Liu, Kun Wang, Jia Luo, Bo Yang and Ruijian He
Slide 1
RESERVOIR SIMULATION GROUP
Outline
• Introduction
• Parallel Reservoir Simulations
• Case Studies
• Conclusions
Slide 2
RESERVOIR SIMULATION GROUP
Introduction
• Desktop/workstation
• Multi-cores, OpenMP (CMG)
• Efficient for small/medium cases
• Long simulation time for large-scale
cases with tens of millions of grid cells
Slide 3
RESERVOIR SIMULATION GROUP
Introduction
• Clusters/supercomputers
• OpenMP for shared memory
- multi-cores, easy to use
• MPI among computation nodes
- communication
- MPI-IO, read input and write output
• Large-scale simulations
Slide 4
RESERVOIR SIMULATION GROUP
Reservoir Simulation
• Reservoir simulators share common modules:
- Grids
- Data, including rock, fluid and other properties
- Linear solvers and preconditioners
- Well modeling and key words parsing
- Input and output
- Visualization
- Memory and communication management
Slide 5
RESERVOIR SIMULATION GROUP
Research Goals
• Develop a common platform for various simulators
-- black oil, compositional and thermal models
-- chemical reaction, combustion
-- geomechnics, etc.
• Simulate extremely large-scale cases with billions of grid cells
• Scalable, thousands of processors
• Investigate parallel linear solvers and preconditioners
• Study parallel computing techniques
Slide 6
RESERVOIR SIMULATION GROUP
Parallel Reservoir Simulations
• Structured hexahedral grids
• Simple and easy to implement
• Finite difference methods
• Finite volume methods
• Corner-point grids (optional)
Slide 7
RESERVOIR SIMULATION GROUP
Parallel Linear Solvers
Model  F(x) = 0  A * x = b
Our platform provides
• Solvers
- Distributed matrices
- GMRES(m), ORTHOMIN(m)
- Distributed vectors
- BiCGSTAB, CG
- Matrix and vector management
• Preconditioners
- Parallel matrix-vector operations
- RAS preconditioner
- Global communication modules
- ILU(k), ILUT and ILUC
- AMG from Hypre
- Constrained pressure residual (CPR)
Slide 8
RESERVOIR SIMULATION GROUP
CPR Preconditioner
• Pressure block App, positive definite
• Saturation block Ass
• Well block Aww
• Coupled matrices
Slide 9
RESERVOIR SIMULATION GROUP
CPR-FPF Preconditioner
Slide 10
RESERVOIR SIMULATION GROUP
Case Studies
• IBM Blue Gene/Q Supercomputer
• IBM US
• PowerA2, 16 (+1) cores, 16 GB mem
• 32,768 CPU cores
Slide 11
RESERVOIR SIMULATION GROUP
Case 1: SPE10, validation
• Water-oil model
• Grid: 60 x 220 x 85, 1.12 million
• Highly heterogeneous permeability
• 2000 days, max step: 100 days
• One injection well, four producers
• Bottom figure: permeability, Kx
• Right figure: oil rate
Slide 12
RESERVOIR SIMULATION GROUP
Case 2: SPE10, three phases, validation
• Standard black oil model using SPE10 geological model
• Grid: 60 x 220 x 85, 1.12 million, 2000 days
• Oil rate of well 1 and well 2
Slide 13
RESERVOIR SIMULATION GROUP
Case 3: validation
• Four components black oil model (API tracking model)
• API tracking case from CMG (data file mxsmo019.dat)
Slide 14
RESERVOIR SIMULATION GROUP
Case 4: refined SPE10
• Oil-water model
• Grid: 300 x 1100 x 425, 140 million
• BiCGSTAB solver
• CPR preconditioner
• 10 days
Slide 15
• Quasi-IMPES decoupling
• Inexact Newton method, tol: 1e-2
• IBM Blue Gene/Q supercomputer
MPI
Steps
Newton
Solver
Time (s)
256
27
102
119
21056.59
512
26
99
114
9321.31
1024
25
90
104
4235.68
2048
25
90
104
2110.16
4096
26
98
124
1265.42
RESERVOIR SIMULATION GROUP
Case 4: refined SPE10, scalability
Slide 16
RESERVOIR SIMULATION GROUP
Case 5: refined SPE1
• Black oil model
• Grid: 100 million
• Inexact Newton method
• tol: 1e-3
Slide 17
• Solver: BiCGSTAB
• Preconditioner: CPR-FPF
• ABF decoupling
• 10 days
RESERVOIR SIMULATION GROUP
Case 5: refined SPE1, scalability
Slide 18
RESERVOIR SIMULATION GROUP
Case 6: Poisson
• 6 billion grid cells
• GMRES(30), 90 iterations
• RAS preconditioner, overlap 1
• Grid generation, build linear system, assemble, and solve
Slide 19
MPI
Grid (s)
Build (s)
Assemble (s) Overall (s)
512
429.78
57.83
130.54
1829.18
1024
200.24
29.28
66.12
906.73
2048
106.83
14.83
34.03
463.59
4096
47.82
7.59
17.70
232.77
RESERVOIR SIMULATION GROUP
Case 6: Poisson, scalability
Slide 20
RESERVOIR SIMULATION GROUP
Conclusions
• A parallel platform has been developed
-- reservoir simulations
-- grids, data, linear solvers, preconditioners
-- well modeling, key word parsing and visualization.
• Our simulators can compute large-scale models.
• The platform and our simulators have excellent scalability.
• Parallel computing is powerful to reservoir simulations.
Slide 21
RESERVOIR SIMULATION GROUP
Sponsors
Slide 22
RESERVOIR SIMULATION GROUP
Thanks.
Slide 23
RESERVOIR SIMULATION GROUP