PACARAIMA, Brazil—Arriving home from work last Friday in this sweltering Amazonian town along Venezuela's border, a shopkeeper was beaten by a group of four men who tied him and his wife up before taking $5,600 he had stuffed in his pocket to buy supplies the next day.

The attack proved to be a tipping point for this impoverished town of 10,000 people, where—like across swaths of South America—sympathy is beginning to give way to resentment over the 2.3 million Venezuelans the U.N. estimates have fled to neighboring countries since 2014.

News of the assault prompted Pacaraima locals, who already had been planning a protest the next day, to set fire to a refugee camp and chase away Venezuelans with rocks. The army said at least 1,200 Venezuelans fled Brazil that day back to their homeland.