Are transvestites former men or women?

Like you know those people who kinda look like girls (some of them really do) but have es. Some you can tell are men and just dressed up but others really look like girls and then they pull their pants down and wham. How can you tell the difference between a women who went through surgery and a man who just dressed up?