There was an article in Newsweek I recently came across that mentions a link between pollution and hormonal levels:
Why Chemicals Called Obesogens May Make You Fat | Newsweek Health | Newsweek.com
Here is a PubMed article about chemical effect on rat hormones:
Pubertal exposure to estrogenic chemicals affects ... [Horm Behav. 2006] - PubMed result
Also, I remember a math class I had in college that mentioned that from the 1950 to the 1990s, sperm count in men had declined something like 40%.
I definitely think that pollution is affecting male hormonal levels, perhaps even in utero. I am sure there is more information on the topic out there.