The team stressed the link was merely "observational", which means they cannot definitively state that air pollution was the cause.
Men exposed to fine particle air pollution may risk having smaller, abnormally-shaped sperm, said a study on Wednesday, warning this "may result in a significant number of couples with infertility".
An analysis of 2001-2014 data for more than 6,400 Taiwanese men and boys aged 15 to 49, found "a robust association" between a decline in "normal" sperm and exposure to PM 2.5 pollution, it said.
PM 2.5 is the term used for air pollution containing the smallest of particles, those measuring 2.5 microns in diameter or less.
A micron is a millionth of a metre.
The link was observed for short-term exposure of three months, as well as for long-term exposure of two years, according to study results published in the medical journal Occupational & Environmental Medicine, though outside specialists questioned the conclusions.
The research team said every increase of five microgrammes per cubic metre of air (5ug/m3) in PM 2.5 exposure over two years, was associated with a "significant drop" of about 1.29 per cent in normal sperm shape and size.
Pollution exposure was measured at each participant's home address using Nasa satellite data.
While sperm shape and size declined, sperm numbers increased, "possibly as a compensatory mechanism", the researchers found.
A similar correlation was witnessed with PM 2.5 exposure of only three months - how long it takes for sperm to be generated.
The team stressed the link was merely "observational", which means they cannot definitively state that air pollution was the cause of sperm size decline.
Allan Pacey, a professor of andrology at the University of Sheffield who was not involved in the study, said sperm size and shape is notoriously hard to assess, and their effect on infertility unclear.
"So, while the authors have found a potentially interesting biological result, I am not sure that it is clinically meaningful."
For Kevin McConway of The Open University in England, other factors not taken into account by the researchers may be responsible for the sperm changes.
"So there has to remain doubt as to whether particulate pollution, or indeed any kind of air pollution, is a cause of semen abnormality," he said via the Science Media Centre in London.
"If I were young enough to worry about my fertility, I wouldn't put moving to an area with cleaner air at the top of my list of actions - though there are certainly many other health-related reasons to live in cleaner air."