Consider using this replay scraper+parser (if it still works):
https://github.com/pts1667/zkscraperThere is a "remote controller" AI as well. You will need to compile for the latest spring version:
https://github.com/pts1667/ControllerAIOfc, this means I've tried something similar in the past.
The biggest issue is you can't effectively use RL or parse replays quickly due to Spring's massive CPU+memory requirements (relative to most other RL environments).
If I were to try this again, I would start with creating a simplified environment (engine, game code, etc.) so that I can run a large amount of RL instances in parallel, mixing in say ~1% actual matches (vs CPU/other AI). I would begin training on raw replays first then proceed with RL in a second training stage.
You will need a LOT of replays. A small amount like 1000, 2000 won't do. You could mix in replays from the simplified environment.
Make sure to apply greater weighting to real replays/RL sims.
Consider fine-tuning an existing LLM model. Not something I've tried, but I'd look into replacing the LM head with dedicated rts output controls, if that makes sense.