The group says new findings about the virus make it "unethical" for the games to go ahead in an open letter to the World Health Organization.
They call on the WHO to revisit its guidance on Zika, which is linked to serious birth defects.
The International Olympic Committee (IOC) said in May it sees no reason to delay or move the games due to Zika.
The outbreak of the mosquito-borne disease began in Brazil a year ago, but now more than 60 countries and territories have continuing transmission.
While Zika's symptoms are mild, in the letter the experts say it causes babies to be born with abnormally small heads and may also cause a rare and sometimes fatal neurological syndrome in adults.
The letter is signed by 150 international scientists, doctors and medical ethicists from such institutions as Oxford University and Harvard and Yale universities in the United States.
They cite the failure of a mosquito-eradication programme in Brazil, and the country's "weakened" health system as reasons to postpone or move the Olympics in "the name of public health".
More on the Zika crisis:
Microcephaly: Why it is not the end of the world
What you need to know Key questions answered about the virus and its spread
Travel advice Countries affected and what you should do
The mosquito behind spread of virus What we know about the insect
Abortion dilemma Laws and practices in Catholic Latin America