Biron, whose long pass set up Stepan's goal, made 20 saves.
Pascal Leclaire stopped 25 shots for Ottawa, which led 1-0, 2-1, 3-2, and 5-4.
White scored short-handed for his second goal of the middle period at 16:33 to put New York up 4-3.
Ruutu, who assisted on Kelly's short-handed goal in the first, tied it at 4 with 30 seconds left in the second.
Rangers enforcer Derek Boogaard was given a major and a game misconduct 12:55 into the first for spearing Ottawa's Chris Neil.
Neil and Milan Michalek left the game in the second with lower body injuries. A Senators spokesman said both players were removed from the game for precautionary reasons.
Kelly, who scored a short-handed goal 9:08 in, got his second goal of the game 1:11 into the second. He got behind Rangers defensemen Steve Eminger and Ryan McDonagh to take Jason Spezza's long pass up the middle and drove in alone on Biron, beating him with a wrist shot into the top right corner.
Copyright by STATS LLC and The Associated Press
Arena: Canadian Tire Centre Location: Ottawa, Ontario
Referees: Dave Jackson, Francis Charron Linesmen: Brad Kovachik, Steve Barton
Attendance: 17,116 (92.5% full)
Chris Kelly 2 (shorthanded) Assist: Jarkko Ruutu
Artem Anisimov 3 Assist: Dale Weise
Chris Kelly 3 Assist: Jason Spezza, Milan Michalek
Todd White 1 Assist: Brandon Dubinsky, Matt Gilroy
Zack Smith 1 Assist: Peter Regin, Jesse Winchester
Derek Stepan 3 Assist: Martin Biron
Todd White 2 (shorthanded) Assist: Ruslan Fedotenko
Jarkko Ruutu 1 Assist: Sergei Gonchar, Ryan Shannon
Zack Smith 2 (power play) Assist: Nick Foligno, Erik Karlsson