It's also interesting to note that the main sword Ardyn is using to fight Noctis with is completely unique to him. I haven't seen a Royal Arm looking like that when I look through Noctis's arsenal of Royal weapons.
Here's what it looks like:
https://i2.wp.com/samurai-gamers.com/wp-content/uploads/2016/11/epilogue.png?resize=662,371&ssl=1
Tbh, I was always under the impression that Ardyn was the first King, since the person on the cover of the Cosmogony looked like him, but this is just speculation of course.
He could have recovered the Royal Arms after he got exiled? I mean nothing is stopping him from going around in his own spare time acquiring them. He got time to kill after all. What else was he gonna do? lol