HadouQen: Adaptive AI Agent Using Reinforcement Learning in Street Fighter II: Special Champion Edition
DOI:
https://doi.org/10.65141/ject.v2i1.n4Keywords:
Proximal Policy Optimization, reinforcement learning, fighting games, Street Fighter II, adaptive AIAbstract
This study presents the development of an AI agent trained using Proximal Policy Optimization (PPO) to compete in Street Fighter II: Special Champion Edition. The agent learned optimal combat strategies through reinforcement learning, processing visual input from frame-stacked grayscale observations (84 × 84 pixels) obtained through the OpenAI Gym Retro environment. Using a convolutional neural network architecture with carefully tuned hyperparameters, the model was trained across 16 parallel environments over 100 million timesteps. The agent was tested against M. Bison, the game's final boss and most challenging opponent, across 1,000 consecutive matches to evaluate performance. Results showed exceptional performance with a 96.7%-win rate and an average reward of 0.912. Training metrics revealed a healthy learning progression, showing steady improvement in average reward per episode, decreased episode length indicating more efficient victories, and stable policy convergence. The findings also demonstrate the effectiveness of PPO-based reinforcement learning in mastering complex fighting game environments and provide a foundation for future research in competitive game-playing agents capable of human-level performance in fast-paced interactive scenarios.
References
Aamir, A., Tamosiunaite, M., & Wörgötter, F. (2021). Caffe2Unity: Immersive visualization and interpretation of deep neural networks. Electronics, 11(1), 83. https://doi.org/10.3390/electronics11010083
Almeida, P., Carvalho, V., & Simões, A. (2024). Reinforcement learning as an approach to train multiplayer first-person shooter game agents. Technologies, 12(3), 34. https://doi.org/10.3390/technologies12030034
Alonso, E., Peter, M., Goumard, D., & Romoff, J. (2021). Deep reinforcement learning for navigation in AAA video games. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21) (pp. 2133–2139). International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2021/294
Andrychowicz, M., Raichuk, A., Sta?czyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., & Bachem, O. (2021). What matters for on-policy deep actor-critic methods? A large-scale study. OpenReview. https://openreview.net/forum?id=nIAxjsniDzg
Ashktorab, Z., Liao, Q. V., Dugan, C., Johnson, J., Pan, Q., Zhang, W., Kumaravel, S., & Campbell, M. (2020). Human-AI collaboration in a cooperative game setting. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 1–20. https://doi.org/10.1145/3415167
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., De Oliveira Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., & Zhang, S. (2019). Dota 2 with large scale deep reinforcement learning. arXiv. https://doi.org/10.48550/arxiv.1912.06680
Clifton, J., & Laber, E. (2020). Q-Learning: Theory and applications. Annual Review of Statistics and Its Application, 7(1), 279–301. https://doi.org/10.1146/annurev-statistics-031219-041220
Dong, S., Wang, P., & Abbas, K. (2021). A survey on deep learning and its applications. Computer Science Review, 40. https://doi.org/10.1016/j.cosrev.2021.100379
Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., & Yannakakis, G. N. (2024). Large language models and games: A survey and roadmap. arXiv. https://arXiv.org/abs/2402.18659
Goldwaser, A., & Thielscher, M. (2020). Deep reinforcement learning for general game playing. In Proceedings of the AAAI Conference on Artificial Intelligence, 34(2), 1701–1708. https://doi.org/10.1609/aaai.v34i02.5533
Halina, E., & Guzdial, M. (2022). Diversity-based deep reinforcement learning towards multidimensional difficulty for fighting game AI. arXiv. https://arXiv.org/abs/2211.02759
Hazra, T., & Anjaria, K. (2022). Applications of game theory in deep learning: A survey. Multimedia Tools and Applications, 81(6), 8963–8994. https://doi.org/10.1007/s11042-022-12153-2
Hu, C., Zhao, Y., Wang, Z., Du, H., & Liu, J. (2023). Games for artificial intelligence research: A review and perspectives. arXiv. https://arXiv.org/abs/2304.13269
Janiesch, C., Zschech, P., & Heinrich, K. (2021). Machine learning and deep learning. Electronic Markets, 31(3), 685–695. https://doi.org/10.1007/s12525-021-00475-2
Jeyakumar, J. V., Noor, J., Cheng, Y., Garcia, L., & Srivastava, M. (2020). How can I explain this to you? An empirical study of deep neural network explanation methods. In Advances in Neural Information Processing Systems (NeurIPS).
Joo, H., & Kim, K. (2019). Visualization of deep reinforcement learning using Grad-CAM: How AI plays Atari games? In 2019 IEEE Conference on Games (CoG) (pp. 1–8). IEEE. https://doi.org/10.1109/cig.2019.8847950
Li, S. E. (2023). Deep reinforcement learning. In Reinforcement Learning for Sequential Decision and Optimal Control (pp. 227–255). Springer. https://doi.org/10.1007/978-981-19-7784-8_10
Osborn, J. C., Lederle-Ensign, D., Wardrip-Fruin, N., & Mateas, M. (2023). Combat in games. eScholarship. https://escholarship.org/uc/item/9zj6r5wz
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Müller, K. (2021). Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3), 247–278. https://doi.org/10.1109/jproc.2021.3060483
Shao, K., Tang, Z., Zhu, Y., Li, N., & Zhao, D. (2019). A survey of deep reinforcement learning in video games. arXiv. https://arXiv.org/abs/1912.10944
Simonov, A., Zagarskikh, A., & Fedorov, V. (2019). Applying behavior characteristics to decision-making process to create believable game AI. Procedia Computer Science, 156, 404–413. https://doi.org/10.1016/j.procs.2019.08.222
Taherdoost, H. (2023). Deep learning and neural networks: Decision-making implications. Symmetry, 15(9), 1723. https://doi.org/10.3390/sym15091723
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019). Human-AI collaboration in data science: Exploring data scientists’ perceptions of automated AI. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), Article 211, 1–24. https://doi.org/10.1145/3359313
Yin, Q., Yang, J., Huang, K., Zhao, M., Ni, W., Liang, B., Huang, Y., Wu, S., & Wang, L. (2023). AI in human-computer gaming: Techniques, challenges and opportunities. Machine Intelligence Research, 20(3), 299–317. https://doi.org/10.1007/s11633-022-1384-6
