Mayer pressured Nadal at the net throughout the match, winning 21 of his 29 approaches.
Nadal saved two set points while serving at 4-5 in the first, but then gave away several crucial points in the tiebreaker, which Mayer clinched with an ace.
Mayer got the first break of the match at 3-3 in the second set when Nadal hit a shot into the top of the net. He converted another break on match point, running down a drop shot and flicking it past Nadal at the net.
The top-seeded Spaniard lost in the third round in Shanghai last year to Austria's Jurgen Melzer, just a few weeks after winning his first U.S. Open title.
It's also just the second time in 16 tournaments this year that he has failed to make the quarterfinals. In August, he fell in the second round of the Canadian Open to Croatia's Ivan Dodig.
After dropping the second set, the second-seeded Scot broke Wawrinka twice to take a 5-0 lead in the third. But Wawrinka battled back to take the next three games before Murray finally closed it out.
"The first set I was up 3-0, two breaks, pretty quickly, and I went up 5-0 in the third set. Both sets got a little bit closer than maybe they needed to be," Murray said.
Roddick is trying to finish his season on a positive note after struggling with injuries and crashing out in the third round of Wimbledon.
After reaching the quarterfinals of the U.S. Open last month, Roddick fell in the first round of the China Open in Beijing and bristled at a question from a reporter about whether he was contemplating retirement.
His play against the 11th-ranked Almagro suggested retirement may be some way off: He won 91 percent of points on his first serve -- 32 of 35 points -- and captured 10 of 12 points at net. He never faced a break point.
Ebden, who had only won eight matches on the ATP tour before Shanghai, closed it out on his fifth match point. Ebden plays Murray next for a spot in the semifinals.
"It will be a tough match," Murray said. "He's obviously just coming off the best win of his career, so he'll be confident."