Looking at Mike's work and develop a chronology, it would suggest that the Vadhagh fled from Corum's plane. The Vadhagh can sometimes flit between planes. In addition, we know that the power of the Vadhagh is built on technology, not magic.
Eventually the Vadhagh came to the Six Worlds of The Dragon in the Sword, Here we find only their females, called the Ghost Women. They tell of leaving their world behind because they fled from the Mabden, but in the process they lost their men. There are even nice connections to the Halflings of The Eternal Champion, since the Eldren are called Ghost Women here, and since they flit between planes.
Interestingly, Mabden is the word used for humanity in the Corum novels, while Eldren is the word used for the elfs in The Eternal Champion.
Sometime afterward these people went to Erekose's World in The Eternal Champion(with their menfolk now rediscovered) taking with them the technology that Erekose later unearths to destroy humanity. The Eldren differ from those other elfin races because they have a very high technology, though one they've put away. Most of the "sorcery" that they have is explained thus.
They were originally a favored race of Arkyn, a God of Law. Thus the move to later become supporters of the Balance and then eventually worshipers of Arioch seems like a slow and inevitable slide, and an ironic one given that it is Arioch who destroys the Vadhagh of Corum's Earth.