Particle accelerators play an important role in a wide range of scientific discoveries and industrial applications. The self-consistent multi-particle simulation based on the particle-in-cell (PIC) method has been used to study charged particle beam dynamics inside those accelerators. However, the PIC simulation is time-consuming and needs to use modern parallel computers for high-resolution applications. In this paper, we implemented a parallel beam dynamics PIC code on multi-node hybrid architecture computers with multiple Graphics Processing Units (GPUs). We used two methods to parallelize the PIC code on multiple GPUs and observed that the replication method is a better choice for moderate problem size and current computer hardware while the domain decomposition method might be a better choice for large problem size and more advanced computer hardware that allows direct communications among multiple GPUs. Using the multi-node hybrid architectures at Oak Ridge Leadership Computing Facility (OLCF), the optimized GPU PIC code achieves a reasonable parallel performance and scales up to 64 GPUs with 16 million particles.
References
[1]
Birdsall, C.K. (1991) Particle-in-Cell Charged-Particle Simulations, Plus Monte Carlo Collisions with Neutral Atoms, PIC-MCC. IEEE Transactions on Plasma Science, 19, 65-85. https://doi.org/10.1109/27.106800
[2]
Friedman, A., Grote, D.P. and Haber, I. (1992) Three-Dimensional Particle Simulation of Heavy-Ion Fusion Beams. Physics of Plasmas, 4, 2203-2210.
https://doi.org/10.1063/1.860024
[3]
Qiang, J., Ryne, R.D., Habib, S. and Decyk, V. (2000) An Object-Oriented Parallel Particle-in-Cell Code for Beam Dynamics Simulation in Linear Accelerators. Journal of Computational Physics, 163, 434-451. https://doi.org/10.1006/jcph.2000.6570
http://www.sciencedirect.com/science/article/pii/S0021999100965707
[4]
Qiang, J., Furman, M.A. and Ryne, R.D. (2004) A Parallel Particle-in-Cell Model for Beam: Beam Interaction in High Energy Ring Colliders. Journal of Computational Physics, 198, 278-294. https://doi.org/10.1016/j.jcp.2004.01.008
[5]
Amundson, J., Spentzouris, P., Qiang, J. and Ryne, R. (2006) Synergia: An Accelerator Modeling Tool with 3-D Space Charge. Journal of Computational Physics, 211, 229-248. http://www.sciencedirect.com/science/article/pii/S0021999105002718
https://doi.org/10.1016/j.jcp.2005.05.024
[6]
Qiang, J., Lidia, S., Ryne, R.D. and Limborg-Deprey, C. (2006) A Three-Dimensional Quasi-Static Model for High Brightness Beam Dynamics Simulation. Physical Review Accelerators and Beams, 9, Article ID: 044204.
https://doi.org/10.1103/PhysRevSTAB.9.044204
[7]
Uriot, D. and Pichoff, N. (2015) Tracewin, CEA Saclay.
[8]
Batygin, Y.K. (2005) Particle-in-Cell Code BEAMPATH for Beam Dynamics Simulations in Linear Accelerators and Beamlines. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 539, 455-489. https://doi.org/10.1016/j.nima.2004.10.029
[9]
Pharr, M. and Fernando, R. (2005) GPU Gems 2: Programming Techniques for High-Performance Graphics and General-Purpose Computation. Addison-Wesley Professional, Boston, MA.
[10]
Owens, J.D., Luebke, D., Govindaraju, N., Harris, M., Krüger, J., Lefohn, A.E. and Purcell, T.J. (2007) A Survey of General-Purpose Computation on Graphics Hardware. Computer Graphics Forum, 26, 80-113.
https://doi.org/10.1111/j.1467-8659.2007.01012.x
[11]
Owens, J.D., Houston, M., Luebke, D., Green, S., Stone, J.E. and Phillips, J.C. (2008) GPU Computing. Proceedings of the IEEE, 96, 879-899.
https://doi.org/10.1109/JPROC.2008.917757
Sanders, J. and Kandrot, E. (2010) CUDA by Example: An Introduction to General-Purpose GPU Programming. Portable Documents, Addison-Wesley Professional, Boston, MA.
[15]
Stantchev, G., Dorland, W. and Gumerov, N. (2008) Fast Parallel Particle-to-Grid Interpolation for Plasma PIC Simulations on the GPU. Journal of Parallel and Distributed Computing, 68, 1339-1349. https://doi.org/10.1016/j.jpdc.2008.05.009
[16]
Burau, H., Widera, R., Honig, W., Juckeland, G., Debus, A., Kluge, T., Schramm, U., Cowan, T.E., Sauerbrey, R. and Bussmann, M. (2010) PIConGPU: A Fully Relativistic Particle-in-Cell Code for a GPU Cluster. IEEE Transactions on Plasma Science, 38, 2831-2839. https://doi.org/10.1109/TPS.2010.2064310
[17]
Decyk, V.K. and Singh, T.V. (2011) Adaptable Particle-in-Cell Algorithms for Graphical Processing Units. Computer Physics Communications, 182, 641-648.
https://doi.org/10.1016/j.cpc.2010.11.009
[18]
Joseph, R.G., Ravunnikutty, G., Ranka, S., D'Azevedo, E. and Klasky, S. (2011) Efficient GPU Implementation for Particle in Cell Algorithm. 2011 IEEE International Parallel & Distributed Processing Symposium, Anchorage, AK, 16-20 May 2011, 395-406. https://doi.org/10.1109/IPDPS.2011.46
[19]
Rossi, F., Londrillo, P., Sgattoni, A., Sinigardi, S. and Turchetti, G. (2012) Towards Robust Algorithms for Current Deposition and Dynamic Load-Balancing in a GPU Particle in Cell Code. AIP Conference Proceedings, 1507, 184-192.
https://doi.org/10.1063/1.4773692
[20]
Bastrakov, S., Donchenko, R., Gonoskov, A., Efimenko, E., Malyshev, A., Meyerov, I. and Surmin, I. (2012) Particle-in-Cell Plasma Simulation on Heterogeneous Cluster Systems. Journal of Computational Science, 3, 474-479.
https://doi.org/10.1016/j.jocs.2012.08.012
[21]
Chen, G., Chacon, L. and Barnes, D.C. (2012) An Efficient Mixed-Precision, Hybrid CPU-GPU Implementation of a Nonlinearly Implicit One-Dimensional Particle-in-Cell Algorithm. Journal of Computational Physics, 231, 5374-5388.
https://doi.org/10.1016/j.jcp.2012.04.040
[22]
Azevedo, E.F.D., et al. (2013) Hybrid MPI/OPENMP/GPU Parallelization of XGC1 Fusion Simulation Code. The International Conference for High-Performance Computing, Networking, Storage and Analysis, Denver, CO, 17-22 November.
[23]
Ibrahim, K.Z., et al. (2013) Analysis and Optimization of Gyrokinetic Toroidal Simulations on Homogenous and Heterogenous Platforms. The International Journal of High Performance Computing Applications, 27, 454-473.
https://doi.org/10.1177/1094342013492446
[24]
Decyk, V.K. and Singh, T.V. (2014) Particle-in-Cell Algorithms for Emerging Computer Architectures. Computer Physics Communications, 185, 708-719.
https://doi.org/10.1016/j.cpc.2013.10.013
[25]
Pang, X. and Rybarcyk, L. (2014) GPU Accelerated Online Multi-Particle Beam Dynamics Simulator for Ion Linear Particle Accelerators. Computer Physics Communications, 185, 744-753. https://doi.org/10.1016/j.cpc.2013.10.033
[26]
Shah, H., Kamaria, S., Markandeya, R., Shah, M. and Chaudhury, B. (2017) A novel Implementation of 2D3V Particle-in-Cell (PIC) Algorithm for Kepler GPU Architecture. Proceedings of 2017 IEEE 24th International Conference on High-Performance Computing (HiPC), 1, 378-387.
[27]
Fatemi, S., Poppe, A.R., Delory, G.T. and Farrell, W.M. (2017) AMITIS: A 3D GPU-Based Hybrid-PIC Model for Space and Plasma Physics. Journal of Physics: Conference Series, 837, Article ID: 012017.
https://doi.org/10.1088/1742-6596/837/1/012017
[28]
Titan (2019) Titan Advancing the Era of Accelerated Computing.
https://www.olcf.ornl.gov/olcf-resources/compute-systems/titan/
Hockney, R.W. and Eastwood, J.W. (1988) Computer Simulation Using Particles. Adam Hilger, New York.
[31]
Netzer, R.H. and Miller, B.P. (1992) What Are Race Conditions?: Some Issues and Formalizations. ACM Letters on Programming Languages and Systems (LOPLAS), 1, 74-88. https://doi.org/10.1145/130616.130623
[32]
Quinn, M.J. (2004) Parallel Programming in C with MPI and Open MP. McGraw-Hill Inc., New York.
[33]
Qiang, J. and Li, X. (2010) Particle-Field Decomposition and Domain Decomposition in Parallel Particle-in-Cell Beam Dynamics Simulation. Computer Physics Communications, 181, 2024-2034. https://doi.org/10.1016/j.cpc.2010.08.021
[34]
Nvidia, C. (2010) Cufft Library.
[35]
Tiwari, D., Gupta, S., Gallarno, G., Rogers, J. and Maxwell, D. (2015) Reliability Lessons Learned from GPU Experience with the Titan Supercomputer at Oak Ridge Leadership Computing Facility. In: Proceedings of the International Conference for High-Performance Computing, Networking, Storage and Analysis, ACM, New York, 38. https://doi.org/10.1145/2807591.2807666
[36]
Wells, J., Bland, B., Nichols, J., Hack, J., Foertter, F., Hagen, G., Maier, T., Ashfaq, M., Messer, B. and Parete-Koon, S. (2016) Announcing Supercomputer Summit. Technical Report, ORNL (Oak Ridge National Laboratory), Oak Ridge, TN.
