I read in a Psychology book somewhere that it was shown through studies that people who were similar in levels of attractiveness had better, fuller and more meaningful relationships. I am assuming this is because it reduces some common insecurities found in couples.
Thats the only thing I can see as being why two people wouldn't "look good together", other than that, just follow your gut. If hes worth it and you guys have enough common ground: I say why not? He could be your soul-mate in disguise. But you'll never know anything till you try!