Baker spent the past three seasons in the same position with St. Louis. He has also worked for Minnesota, Detroit and San Diego coaching the defensive line and linebackers.
Baker replaced Sal Sunseri, who left last month to become an assistant at Alabama. Sunseri was one of four defensive aides to leave Carolina.
The Panthers have since hired Ron Meeks as defensive coordinator, Richard Smith as linebackers coach and Ron Milus to coach the secondary.
They'll try to shore up a defense that allowed 30 or more points in six of its final seven games, including a 33-13 loss to Arizona in the playoffs.