ANALYSIS OF PENETRATION TESTING AUTOMATION USING MARKOV DECISION PROCESSES

DOI: 10.31673/2409-7292.2025.017625

Abstract

With the development of cyber threats, penetration testing is becoming critically important for ensuring information
security. The article investigates the automation of this process using Markov decision processes and the Q-learning algorithm.
The application of MDP allows you to model attack scenarios, predict risks, and automate decision-making in stochastic
environments. The main components of the study are the formulation of a Markov environment, the creation of an algorithm
for analyzing the path to vulnerabilities, and the implementation of an interactive web application that integrates with modern
technologies such as Spring Boot, React, and MySQL. The proposed tool models the vulnerability search process, optimizing
it through the Q-learning algorithm that determines optimal policies. Integration with cloud platforms provides scalability and
ease of use. Experimental results confirm the effectiveness of the proposed approach, in particular, reducing testing time,
increasing the accuracy and adaptability of the system. The article analyzes other modern research in the field of pentest
automation, focusing on the use of deep reinforcement learning and graph attack models. The paper discusses limitations, in
particular the need for significant computing resources, and suggests ways to overcome them, for example, training algorithms
based on real user data. Overall, the study demonstrates the high potential of penetration testing automation, contributing to
increasing the accuracy of information systems analysis and their security. In the future, it is planned to optimize the training
algorithms, integrate new data sources, such as CVE reports and bug bounty platforms, which will help expand the functionality
of the tool.
Keywords: Markov decision processes, Artificial Intelligence, cybersecurity.

References
1. Tolkachova, A., & Piskozub, A. (2024). Methods for testing the security of web applications. Cybersecurity:
Education, Science, Technique, 2(26), 115–122. https://doi.org/10.28925/2663-4023.2024.26.668
2. Gore, R., Padilla, J., & Diallo, S. (2017). Markov chain modeling of cyber threats. The Journal of Defense
Modeling and Simulation: Applications, Methodology, Technology, 14(3), 233–244. https://doi.org/10.1177
/1548512916683451
3. Wang, Y., Li, Y., Xiong, X., Zhang, J., Yao, Q., & Shen, C. (2023). DQfD-AIPT: An intelligent penetration
testing framework incorporating expert demonstration data. Security and Communication Networks, 2023, 1–15.
https://doi.org/10.1155/2023/5834434
4. Yi, J., & Liu, X. (2023). Deep reinforcement learning for intelligent penetration testing path design. Applied
Sciences, 13(16), 9467. https://doi.org/10.3390/app13169467
5. Cody, T. (2022). A layered reference model for penetration testing with reinforcement learning and attack
graphs. In 2022 IEEE 29th Annual Software Technology Conference (STC). IEEE. https://doi.org/10.1109
/stc55697.2022.00015
6. Tolkachova, A., & Posuvailo, M.-M. (2024). Penetration testing using deep reinforcement learning.
Cybersecurity: Education, Science, Technique, 17–30. https://doi.org/10.28925/2663-4023.2024.23.1730
7. Spring Boot. (n.d.). Spring Boot. Retrieved from https://spring.io/projects/spring-boot
8. React. (n.d.). React. Retrieved from https://react.dev/
9. Cloud Application Platform | Heroku. (n.d.). Cloud Application Platform | Heroku. Retrieved from
https://www.heroku.com/
10. MySQL. (n.d.). Retrieved from https://www.mysql.com/
11. CVE - CVE. (n.d.). CVE - CVE. Retrieved from https://cve.mitre.org/
12. Unsupported Browser | HackerOne. (n.d.). HackerOne | #1 Trusted Security Platform and Hacker Program.
Retrieved from https://hackerone.com/bug-bounty-programs