Wait a second. Are you saying that you KNEW he flopped a small flush when you called his raise? There’s lots of other hands he could be raising with, including top pair with a good flush draw, a small set, or another overpair. You have to play against his whole range, not just what he ended up having.
As it stands, I think your calling him down is okay, especially once you turned a set of queens. Sometimes these aggressive guys catch the flops that they need, and they usually get paid off when they do.