People indeed get them from the Rohan raids, but there are often quite some available in the auction house, or you can ask in the global channel if anyone has any for sale (i've also often seen people selling them in the glff channel). Prices may vary on the different servers. I have found on my server (Windfola) that the wyrmfire's are often sold for the most gold (50-70g) and that the other gems range from 15-50g. Sometimes people even give them away or sell them for just a few silver on AH. If you often check AH and pay attention to glff I'm sure you will find some for a nice price.
Edit: in AH I believe they are listed under the 'barter items' tab