Python Parallel Processing and Multiprocessing: A Rivew

Authors

  • Zina A. Aziz Technical College of Information Akre, Duhok Polytechnic University, Kurdistan Region-Iraq
  • Diler Naseradeen Abdulqader Department of Computer and Communication, Nawroz University, Kurdistan Region-Iraq
  • Amira Bibo Sallow Department of Computer and Communication, Nawroz University, Kurdistan Region-Iraq
  • Herman Khalid Omer Department of Computer and Communication, Nawroz University, Kurdistan Region-Iraq

DOI:

https://doi.org/10.25007/ajnu.v10n3a1145

Abstract

Parallel and multiprocessing algorithms break down significant numerical problems into smaller subtasks, reducing the total computing time on multiprocessor and multicore computers. Parallel programming is well supported in proven programming languages such as C and Python, which are well suited to “heavy-duty” computational tasks. Historically, Python has been regarded as a strong supporter of parallel programming due to the global interpreter lock (GIL). However, times have changed. Parallel programming in Python is supported by the creation of a diverse set of libraries and packages. This review focused on Python libraries that support parallel processing and multiprocessing, intending to accelerate computation in various fields, including multimedia, attack detection, supercomputers, and genetic algorithms. Furthermore, we discussed some Python libraries that can be used for this purpose.

Downloads

Download data is not yet available.

References

S. M. Lim, A. B. M. Sultan, M. N. Sulaiman, A. Mustapha, and K. Y. Leong, “Crossover and mutation operators of genetic algorithms,” Int. J. Mach. Learn. Comput., vol. 7, no. 1, pp. 9–12, 2017, doi: 10.18178/ijmlc.2017.7.1.611.

I. V. Kotenko, I. B. Saenko, and A. G. Kushnerevich, “Architecture of the parallel big data processing system for security monitoring of internet of things networks,” SPIIRAS Proc., vol. 4, no. 59, pp. 5–30, 2018, doi: 10.15622/sp.59.1.

A. Ebrahim, J. A. Lerman, B. O. Palsson, and D. R. Hyduke, “COBRApy: COnstraints-Based Reconstruction and Analysis for Python,” BMC Syst. Biol., vol. 7, 2013, doi: 10.1186/1752-0509-7-74.

L. D. Dalcin, R. R. Paz, P. A. Kler, and A. Cosimo, “Parallel distributed computing using Python,” Adv. Water Resour., vol. 34, no. 9, pp. 1124–1139, 2011, doi: 10.1016/j.advwatres.2011.04.013.

A. Rogohult, “Benchmarking Python Interpreters,” KTH, Sk. för datavetenskap och Kommun. (CSC), 2016, p. 119, 2016.

W. Y. Je, S. Teh, and H. R. Chern, “A parallelizable chaos-based true random number generator,” 2018.

W. K. Lee, R. C. W. Phan, W. S. Yap, and B. M. Goi, “SPRING: a novel parallel chaos-based image encryption scheme,” Nonlinear Dyn., vol. 92, no. 2, pp. 575–593, 2018, doi: 10.1007/s11071-018-4076-6.

T. Wang and Q. Kemao, “Parallel computing in experimental mechanics and optical measurement: A review (II),” Opt. Lasers Eng., vol. 104, no. June 2017, pp. 181–191, 2018, doi: 10.1016/j.optlaseng.2017.06.002.

E. Tejedor et al., “PyCOMPSs: Parallel computational workflows in Python,” Int. J. High Perform. Comput. Appl., vol. 31, no. 1, pp. 66–82, 2017, doi: 10.1177/1094342015594678.

R. Filguiera, A. Krause, M. Atkinson, I. Klampanos, and A. Moreno, “Dispel4py: A Python framework for data-intensive scientific computing,” Int. J. High Perform. Comput. Appl., vol. 31, no. 4, pp. 316–334, 2017, doi: 10.1177/1094342016649766.

S. R. M. Zebari and N. O. Yaseen, “Effects of Parallel Processing Implementation on Balanced Load-Division Depending on Distributed Memory Systems Client / Server Principles,” vol. 5, no. 3, 2011.

M. Wilde, M. Hategan, J. M. Wozniak, B. Clifford, D. S. Katz, and I. Foster, “Swift: A language for distributed parallel scripting,” Parallel Comput., vol. 37, no. 9, pp. 633–652, 2011, doi: 10.1016/j.parco.2011.05.005.

S. Y. Yu, S. R. Chhetri, A. Canedo, P. Goyal, and M. A. Al Faruque, “Pykg2vec: A python library for knowledge graph embedding,” arXiv, vol. 22, pp. 1–6, 2019.

C. Evangelinos and C. Hill, “Cloud Computing for Parallel Scientific HPC Applications: Feasibility of Running Coupled Atmosphere-Ocean Climate Models on Amazon’s EC2,” Ratio, vol. 2, Jan. 2008.

K. Asanovíc et al., “The Landscape of Parallel Computing Research : A View from Berkeley,” pp. 1–54, 2006.

T. Kim, “Survey and Performance Test of Python-based Libraries for Parallel Processing,” pp. 1–4, 2020.

N. Singh, L. M. Browne, and R. Butler, “Parallel astronomical data processing with Python: Recipes for multicore machines,” Astron. Comput., vol. 2, pp. 1–10, 2013, doi: 10.1016/j.ascom.2013.04.002.

B. Lewis, I. Smith, M. Fowler, and J. Licato, “The robot mafia: A test environment for deceptive robots,” 28th Mod. Artif. Intell. Cogn. Sci. Conf. MAICS 2017, pp. 189–190, 2017, doi: 10.1145/1235.

Y. Babuji et al., “Introducing Parsl: A Python Parallel Scripting Library,” pp. 1–2, 2017, [Online]. Available: https://doi.org/10.5281/zenodo.891533#.WdOPKS_nvdE.mendeley.

P. Moritz et al., “Ray : A Distributed Framework for Emerging AI Applications This paper is included in the Proceedings of the,” USENIX Symp. Oper. Syst. Des. Implement., 2018.

J. Sampé, G. Vernik, M. Sánchez-Artigas, and P. García-López, “Serverless data analytics in the IBM cloud,” Middlew. Ind. 2018 - Proc. 2018 ACM/IFIP/USENIX Middlew. Conf. (Industrial Track), no. October 2019, pp. 1–8, 2018, doi: 10.1145/3284028.3284029.

M. Jaxa-Rozen and J. Kwakkel, “PyNetLogo: Linking NetLogo with Python,” J. Artif. Soc. Soc. Simul., vol. 21, Mar. 2018, doi: 10.18564/jasss.3668.

D. Meunier et al., “NeuroPycon: An open-source python toolbox for fast multi-modal and reproducible brain connectivity pipelines,” Neuroimage, vol. 219, no. June, 2020, doi: 10.1016/j.neuroimage.2020.117020.

J. Kready, S. A. Shimray, M. N. Hussain, and N. Agarwal, “YouTube data collection using parallel processing,” Proc. - 2020 IEEE 34th Int. Parallel Distrib. Process. Symp. Work. IPDPSW 2020, pp. 1119–1122, 2020, doi: 10.1109/IPDPSW50202.2020.00185.

J. Niruthika and S. Pranavan, “222 implementation of parallel aho-corasick algorithm in python,” no. November, pp. 222–233, 2019.

A. Benítez-Hidalgo, A. J. Nebro, J. García-Nieto, I. Oregi, and J. Del Ser, “jMetalPy: A python framework for multi-objective optimization with metaheuristics,” arXiv, 2019.

Y. Babuji et al., “Parsl: Pervasive parallel programming in Python,” HPDC 2019- Proc. 28th Int. Symp. High-Performance Parallel Distrib. Comput., pp. 25–36, 2019, doi: 10.1145/3307681.3325400.

H. Han, “BayesFactorFMRI: Implementing Bayesian Second-Level fMRI Analysis with Multiple Comparison Correction and Bayesian Meta-Analysis of fMRI Images with Multiprocessing,” J. Open Res. Softw., vol. 9, no. 1, pp. 1–7, 2021, doi: 10.5334/jors.328.

G. Heine, T. Woltron, and W. Alexander, “Towards a Scalable Data-Intensive Text Processing Architecture with Python and Towards a Scalable Data-Intensive Text Processing Architecture with Python and Cassandra,” no. November, 2018.

D. Datta, D. Mittal, N. P. Mathew, and J. Sairabanu, “Comparison of Performance of Parallel Computation of CPU Cores on CNN model,” Int. Conf. Emerg. Trends Inf. Technol. Eng. ic-ETITE 2020, pp. 1–8, 2020, doi: 10.1109/ic-ETITE47903.2020.142.

T. Shaffer, Z. Li, B. Tovar, and Y. Babuji, “Lightweight Function Monitors for Fine-Grained Management in Large Scale Python Applications,” pp. 1–11.

H. Park, J. Denio, J. Choi, and H. Lee, “MpiPython: A robust python MPI binding,” Proc. - 3rd Int. Conf. Inf. Comput. Technol. ICICT 2020, pp. 96–101, 2020, doi: 10.1109/ICICT50521.2020.00023.

J. J. Galvez, K. Senthil, and L. Kale, “CharmPy: A Python Parallel Programming Model,” Proc. - IEEE Int. Conf. Clust. Comput. ICCC, vol. 2018-Septe, pp. 423–433, 2018, doi: 10.1109/CLUSTER.2018.00059.

R. Eggen and E. M. Eggen, “Thread and Process Efficiency in Python,” pp. 32–36.

M. R. Rizqullah, A. R. Anom Besari, I. Kurnianto Wibowo, R. Setiawan, and D. Agata, “Design and implementation of middleware system for IoT devices based on raspberry Pi,” Int. Electron. Symp. Knowl. Creat. Intell. Comput. IES-KCIC 2018 - Proc., pp. 229–234, 2019, doi: 10.1109/KCIC.2018.8628528.

V. Skorpil, V. Oujezsky, P. Cika, and M. Tuleja, “Parallel Processing of Genetic Algorithms in Python Language,” Prog. Electromagn. Res. Symp., vol. 2019-June, pp. 3727–3731, 2019, doi: 10.1109/PIERS-Spring46901.2019.9017332.

H. Jan, J. H. Pynetlogo, L. Netlogo, M. Jaxa-rozen, and J. H. Kwakkel, “Article PyNetLogo : Linking NetLogo with Python Reference PyNetLogo : Linking NetLogo with Python,” vol. 21, no. 2.

P. Zhang, Y. Gao, and X. Shi, “QuantCloud: A software with automated parallel python for Quantitative Finance applications,” Proc. - 2018 IEEE 18th Int. Conf. Softw. Qual. Reliab. Secur. QRS 2018, pp. 388–396, 2018, doi: 10.1109/QRS.2018.00052.

V. Sindhu, “ScholarWorks @ Georgia State University Exploring Parallel Efficiency and Synergy for Max-P Region Problem Using Python,” 2018.

F. Real, A. Batou, T. Ritto, and C. Desceliers, “Stochastic modeling for hysteretic bit–rock interaction of a drill string under torsional vibrations,” J. Vib. Control, p. 107754631982824, 2019, doi: 10.1177/ToBeAssigned.

Z. Rinkevicius et al., “VeloxChem: A Python-driven density-functional theory program for spectroscopy simulations in high-performance computing environments,” Wiley Interdiscip. Rev. Comput. Mol. Sci., vol. 10, no. 5, pp. 1–14, 2020, doi: 10.1002/wcms.1457.

V. Canh Vu and T. H. Hoang, “Detect Wi-Fi Network Attacks Using Parallel Genetic Programming,” Proc. 2018 10th Int. Conf. Knowl. Syst. Eng. KSE 2018, pp. 370–375, 2018, doi: 10.1109/KSE.2018.8573378.

S. Khan and A. Latif, “Python based scenario design and parallel simulation method for transient rotor angle stability assessment in PowerFactory,” 2019 IEEE Milan PowerTech, PowerTech 2019, pp. 1–6, 2019, doi: 10.1109/PTC.2019.8810949.

A. V. M. Barone and R. Sennrich, “A parallel corpus of Python functions and documentation strings for automated code documentation and code generation,” arXiv, 2017.

Downloads

Published

2021-08-30

How to Cite

A. Aziz, Z., Naseradeen Abdulqader, D., Sallow, A. B., & Khalid Omer, H. (2021). Python Parallel Processing and Multiprocessing: A Rivew. Academic Journal of Nawroz University, 10(3), 345–354. https://doi.org/10.25007/ajnu.v10n3a1145

Issue

Vol. 10 No. 3 (2021)

Section

Articles

License

Copyright (c) 2021 Zina A. Aziz, Diler Naseradeen Abdulqader, Amira Bibo Sallow, Herman Khalid Omer

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Authors retain copyright

The use of a Creative Commons License enables authors/editors to retain copyright to their work. Publications can be reused and redistributed as long as the original author is correctly attributed.

 

  1. Copyright

 

  1. The researcher(s), whether a single or joint research paper, must sell and transfer to the publisher (the Academic Journal of Nawroz University) through all the duration of the publication which starts from the date of entering this Agreement into force, the exclusive rights of the research paper/article. These rights include the translation, reuse of papers/articles, transmit or distribute, or use the material or parts(s) contained therein to be published in scientific, academic, technical, professional journals or any other periodicals including any other works derived from them, all over the world, in English and Arabic, whether in print or in electronic edition of such journals and periodicals in all types of media or formats now or that may exist in the future. Rights also include giving license (or granting permission) to a third party to use the materials and any other works derived from them and publish them in such journals and periodicals all over the world. Transfer right under this Agreement includes the right to modify such materials to be used with computer systems and software, or to reproduce or publish it in e-formats and also to incorporate them into retrieval systems.

 

  1. Reproduction, reference, transmission, distribution or any other use of the content, or any parts of the subjects included in that content in any manner permitted by this Agreement, must be accompanied by mentioning the source which is (the Academic Journal of Nawroz University) and the publisher in addition to the title of the article, the name of the author (or co-authors), journal’s name, volume or issue, publisher's copyright, and publication year.

 

  1. The Academic Journal of Nawroz University reserves all rights to publish research papers/articles issued under a “Creative Commons License (CC BY-NC-ND 4.0) which permits unrestricted use, distribution, and reproduction of the paper/article by any means, provided that the original work is correctly cited.

 

  1. Reservation of Rights

The researcher(s) preserves all intellectual property rights (except for the one transferred to the publisher under this Agreement).

  1. Researcher’s guarantee

The researcher(s) hereby guarantees that the content of the paper/article is original. It has been submitted only to the Academic Journal of Nawroz University and has not been previously published by any other party.

In the event that the paper/article is written jointly with other researchers, the researcher guarantees that he/she has informed the other co-authors about the terms of this agreement, as well as obtaining their signature or written permission to sign on their behalf.

The author further guarantees:

  • The research paper/article does not contain any defamatory statements or illegal comments.
  • The research paper/article does not violate other's rights (including but not limited to copyright, patent, and trademark rights).

This research paper/article does not contain any facts or instructions that could cause damages or harm to others, and publishing it does not lead to disclosure of any confidential information.