Autonomous Mathematical Discovery in an Open-World Multi-Agent Environment