you should be able to run both cards on one machine no problem, just set the settings for each one.
Or if you want them out of the case to get better air flow you can buy a pci-e riser card.

There is still a lot to learn. For the 580, make sure it is in compute mode. You can google instructions for that in the ati software. According to, you should be getting more than you are.
Also be careful with running the AMD RX card and the GTX on the same motherboard. I have heard stories about the RX card drawing too much power from the board and you'll have some burned 24p PSU cables.
Mining does come with additional costs. But if you can mine successfully, you will be able to cover all that you have spent.