Henrik Lundqvist stopped 29 shots for New York (8-3-3), which hit the road for the first time since a 4-1-1 homestand. The Rangers haven't lost since Oct. 29 when Ottawa rallied from three goals down in the third period and won 5-4 in a shootout.
Gaborik, who extended his point streak to four games, and Stepan scored 2:17 apart to put the Rangers up 2-1 midway through the second period.
Stepan assisted on both of Gaborik's goals, including the Slovakian right wing's second of the game that gave New York a short-lived, two-goal lead 8:30 into the third. Artem Anisimov earned his second assist of the game on the goal.
The line combined for seven points.
"I think we feel pretty good out there," Gaborik said. "Arty and Step are playing with a lot of confidence. We scored a couple of goals off great forechecks by those guys. Hopefully we can keep it up."
Rangers coach John Tortorella didn't have any quick answers why this trio has clicked. The belief was that Gaborik would play on a line with prized free-agent center Brad Richards, who was signed this summer. Early trials together failed to produce the expected chemistry.
"You always try to put lines together and see if they work, and sometimes lines look good on paper but they don't show up in the games," Tortorella said. "So you try different things. We were struggling a little bit. We were a little stale offensively. We tried some different things and that line has worked."
Nick Foligno drew Ottawa within a goal 1:18 later, but the Senators' comeback attempt fell short. Milan Michalek scored his 10th goal and Craig Anderson made 16 saves for Ottawa, which lost its fourth in a row.
Senators captain Daniel Alfredsson missed his fifth straight game because of a concussion. Ottawa is 1-3-1 since losing Alfredsson, who was injured on a hit to the head by Rangers forward Wojtek Wolski in the previous meeting between the teams.
The Rangers' fortunes have headed in the opposite direction since the hit, which drew a minor penalty for Wolski but no supplemental discipline.
Anisimov had a breakaway in the first but Anderson got a piece of his shot, which glanced off the left post.
Ottawa took a 1-0 lead 8:39 into the second.
Jason Spezza stick-handled his way past referee Ghislain Hebert to start the scoring play before Michalek picked up the puck in the slot and put a wrist shot past Lundqvist.
The Rangers struck back to tie it and take the lead with two quick goals, both resulting from turnovers by Ottawa defensemen.
Stepan intercepted Sergei Gonchar's pass back along the left boards to set up the tying goal. He fed a pass to Gaborik, who put a shot past Anderson at 9:21.
Anisimov cut off David Rundblad's misguided pass attempt as the rookie defenseman tried to carry the puck out from behind the net. Stepan jumped on the loose puck to beat a helpless Anderson for his third goal at 11:38.
"I think they maybe lost their focus a little bit after their first goal and we definitely made them pay a little bit," Lundqvist said.
Stepan fed a cross-ice pass to Gaborik to launch the 2-on-1 break that led to New York's third goal. Gaborik fired a shot past Anderson from the left circle for his ninth goal and second of the game.
"I think they've found a chemistry since they've started playing together," Gonchar said. "They're creating a lot of scoring chances. It's going well for them, but I don't think that's an excuse for us to give them all those chances and make those turnovers."
Gaborik has four goals and three assists during his point streak. Stepan and Anisimov both have points in three consecutive games. ... Lundqvist made a fantastic glove save to rob Senators D Erik Karlsson 3:08 into the second. ... Karlsson earned his team-leading 13th assist on Michalek's goal.