The team estimated Pierre-Paul would be recovered from the surgery in 12 weeks.
Pierre-Paul, 24, played with the back injury last year and received multiple epidural injections to get through the season.
"My back has been bothering me for a while, and even last season I played through some back issues," Pierre-Paul said in statement from the team.
"The treatment I've been doing has helped manage the issue, but this procedure is going to get me fully healthy and there should be no more pain. I'm going to do everything the doctors tell me to do during my rehab so I can be back playing the game I love at a high level as quickly as possible."
The belief is he should be ready for the start of the 2013 season and should be 100 percent, a source said.
The surgery was being performed by Dr. Robert Watkins.
"The timing of his return to football will depend on his response to the rehabilitation program. We anticipate a full recovery," Watkins said in a statement.
Pierre-Paul said in April that he was not out of shape last season when his production dipped from 16.5 sacks in 2011 to 6.5 sacks in 2012. The Pro Bowl pass-rusher was responding to comments made earlier in the offseason by coach Tom Coughlin.
The coach said at the owners meeting in Arizona, "He didn't play as well as the year before. … He was big when he came to camp. There's not a lot of body fat [with Pierre-Paul], yes, but still."
"I was always in great shape," Pierre-Paul said. "I'm still the same size this year. People say I look slimmer but I'm the same weight.
"I was not big last year," he added. "I was 273 and I'm 274 now. If they want me to lose weight, I can do it that fast."
Justin Tuck thought Pierre-Paul looked slimmer this offseason.
"He looks slim," Tuck said about his teammate after an offseason workout in April. "Just the first two days of running and working and lifting, he looks primed to have another great year for us, and we are going to need him because he is a big part of our success."
ESPNNewYork.com's Ohm Youngmisuk contributed to this report.