Down 3-1 in the eighth, Baltimore got within a run on a pinch-hit RBI single by Felix Pie. One inning later, Scott hit a 1-0 pitch from Rivera over the 25-foot scoreboard in right field.
It was the 27th homer of the season for Scott, but his first this month. He entered 1 for 12 lifetime against Rivera.
"Considering the situation, the best closer I think that's ever taken the mound, it's just another experience that I'll never forget," said Scott, who entered as a pinch hitter in the seventh inning. "I'm thankful for the opportunity and the way things worked out."
Scott led off the 11th with a double against David Robertson (4-5). Ty Wigginton followed with a long single to right-center, scoring Scott without a throw and providing the Orioles with their eighth straight extra-inning win.
"The way I was looking at it was, I've got to get him to third base regardless," Wigginton said. "He left me a breaking ball in the zone, and I was able to hit it in the gap."
Mike Gonzalez (1-3) got the victory with one hectic inning of work.
Pettitte returned from a two-month stay on the disabled list to allow one run, three hits and a walk in six innings. The left-hander, who had been sidelined since July 18 with a strained left groin, retired his final 11 batters.
"I didn't feel great the first couple of innings, but as the game went on I got comfortable with my mechanics," Pettitte said. "I found my release point a little better on my fastball, just moving the ball around the way I wanted to."
Yankees manager Joe Girardi also was encouraged.
"He did a great job," Girardi said. "I could tell he was gone for two months. I missed him."
Pettitte left with New York up 3-1. Rivera's fourth blown save deprived Pettitte of the victory, but the veteran starter did not complain.
"It's very rare when you see Mo blow a save," Pettitte said. "You're just reminded he is human and it's not automatic."
After missing 57 games, Pettitte is back in time to contribute to the Yankees' push to win the division crown. New York has a half-game lead over Tampa Bay, which opens a four-game series at Yankee Stadium on Monday night, when New York will honor late owner George Steinbrenner in Monument Park.
Girardi planned to rest first baseman Mark Teixeira and third baseman Alex Rodriguez, but both were summoned as pinch hitters in the 11th. Rodriguez drew a leadoff walk and was replaced by pinch-runner Eduardo Nunez, who advanced to third when Gonzalez threw wildly off his back foot toward first base.
"I needed a ground-ball double play right there," Gonzalez said. "I wasn't looking at anything else but that."
The left-hander wasn't about to let the game get away after Scott's ninth-inning homer.
"As soon as he hit that, we knew we were going to win," Gonzalez said.
New York now heads home after going 3-6 on a swing through Texas, Tampa Bay and Baltimore.
"This was a tough road trip," Girardi said. "We had some losses like this. We have to go home, and that's a good thing."
The Yankees got a first-inning run against Chris Tillman when Brett Gardner walked and scored on a two-out single by Robinson Cano. In the bottom half, cleanup hitter Adam Jones drove in a run with a two-out bunt.
New York used four walks and a single to go up 3-1 in the fourth. After Cano led off with a four-pitch walk, Nick Swisher struck out and Jorge Posada walked before Curtis Granderson lined an RBI single up the middle.
Cano has a 12-game hitting streak against Baltimore. ... New York won the season series 13-5. ... Tillman issued six walks, tying a career high. ... The Orioles improved to 27-17 under manager Buck Showalter.