He loves his family more.
So when his football family said take unlimited time off two weeks before the season to be with his son, T.J., as he underwent open heart surgery for the third time since being born with a severe congenital heart defect in 2012, it was a relief.
"It's made life a lot easier, and it's not something I take for granted," Olsen said on Monday after returning to practice for the first time in seven days. "You would hope that it would be this way everywhere, but sometimes people aren't quite as understanding."
The Panthers have been. Team owner Jerry Richardson, who received a heart transplant in 2009, flew Olsen, his wife Kara, T.J., and other family members to Boston in 2012 to consult with doctors about an experimental surgery.
Richardson actually made the trip and spent time in hospital waiting rooms with Olsen.
"We talk about family and we want to follow up on that," Panthers coach Ron Rivera said. "Mr. Richardson has fostered a great environment here for us and we've taken the ball and run with it."
Rivera would like to believe all 32 NFL teams take the same approach. But when asked if other organizations he's been around have, he said, "Well, I just know this, I'm more involved in it a little bit more. That's all I'll say about that."
There are other examples where the Panthers put family first. Rivera made time before practice last Wednesday to play catcher for his daughter, a pitcher at UCLA, before she returned to college.
Quarterback Derek Anderson was given time off last week to be with his wife as she gave birth to the couple's first child even though it left third-stringer Joe Webb as the only healthy quarterback, as starter Cam Newton was out with fractured ribs and fourth-stringer Matt Blanchard was on injured reserve with a concussion.
"It all worked out," said Olsen, who happened to be at the same hospital as Anderson. "DA got to be there for his wife and [at Thursday's exhibition finale at Pittsburgh]. I didn't miss much. It all works out if you do things the right way."
Olsen would like to think the family appraoch has helped make the Carolina locker room and team stronger as it attempts to record consecutive winning seasons for the first time in franchise history.
"It's a special group," he said. "I've said that since I got here."
Olsen is happy to report T.J. is recovering well. But because his son remains in intensive care and is maybe weeks from being discharged, Olsen goes back and forth between the hospital and the stadium in his free time.
"He's a tough little guy," Olsen said. "He's really responded well to all three surgeries. We've had a few little hiccups, but for the most part he's on the right track."
And while there are no guarantees there won't be a transplant or other surgeries in T.J.'s future, Olsen has the support of the organization and is able to focus on football and Sunday's opener at Tampa Bay without added stress.
And the Panthers will need Olsen. He led the team in catches last season with 73. He is a big part of a two-tight end set Carolina plans to implement in an attempt to diversify the offense and take pressure off a new group of wide receivers.
Olsen vows he'll be ready.
"I haven't felt a ton of pressure to be two places at once, and I've been able to handle my family and that priority first," he said. "But also realize, this is a priority for me, too. It's important for me to be here.
"It won't be any challenge. I'll make sure I get what I have to get done. My wife is pretty understanding. She gets it."
So, apparently, do the Panthers.