The team said Pagano will be introduced at a news conference on Thursday. It will be the first head coaching job for the 51-year-old Pagano, who has been a career assistant until now with stops in Oakland and Cleveland in the NFL and stints at schools including Miami and North Carolina.
"It's difficult to leave the Ravens but I couldn't pass up on this great opportunity," Pagano said in a statement released by the Ravens. "I'm just thrilled and so excited."
Pagano interviewed with the Colts on Tuesday after Baltimore was eliminated from the playoffs Sunday in a 23-20 loss to the New England Patriots in the AFC Championship Game.
This season was Pagano's first in charge of the vaunted Ravens defense. Pagano followed Greg Mattison, who left Baltimore after two seasons as defensive coordinator for the same position at the University of Michigan in January 2011.
Under Pagano, Baltimore allowed the third-fewest points and yards in the league in 2011. Safety Ed Reed, linebackers Ray Lewis and Terrell Suggs and nose tackle Haloti Ngata also made the Pro Bowl under Pagano.
"I like it," Pro Bowl defensive end Robert Mathis wrote on Twitter.
Pagano replaces Jim Caldwell, who was fired after the Colts' 2-14 season in which quarterback Peyton Manning never played a down as he recovered from neck surgery.
The move is just the latest in a dizzying series of changes by owner Jim Irsay.
The Colts fired Caldwell last week after three seasons. The team went to the Super Bowl during Caldwell's first year, but this year locked up the No. 1 overall draft pick with a horrid performance that also cost team vice chairman Bill Polian and his son, general manager Chris, their jobs.
SportsNation: Colts hire Chuck Pagano
Does hiring defensive-minded Chuck Pagano as head coach suggest the end of Peyton Manning era for the Colts? Vote!
Irsay has since hired 39-year-old Ryan Grigson as the new GM while letting go of Caldwell's staff. In all, 11 of the 20 coaches who started the season are gone and others may go, too, once Pagano arrives.
Pagano is the fourth former Ravens defensive coordinator to land a head coaching job, following Marvin Lewis, Mike Nolan and Rex Ryan.
The Wyoming graduate and former strong safety for the Cowboys began his coaching career in 1984 as a graduate assistant at Southern California and spent time in the college ranks at Boise State, UNLV, East Carolina and Miami before joining Cleveland to coach the secondary. In 2005-06, he was defensive backs coach at Oakland, then served as defensive coordinator at North Carolina before joining the Ravens when John Harbaugh became head coach four years ago.
"Chuck is unorthodox," Suggs said. "He's like the Joker. You never really expect what he's going to do, and everything has a motive."
The Ravens considered Pagano to be just one of the guys.
"What makes him good? He relates to the players a whole lot," defensive end Cory Redding said. "He's almost like a player in a D-coordinator's position. The guy has so much fun with us. He treats you like more than a player. It's like we're his sons. He wants us to do well. He keeps it fresh. He knows everybody's strengths and puts them in position to make plays."
Asked last month if he had aspirations to be a head coach, Pagano replied, "When I was a kid growing up, my dad being a football coach, he asked the same question of all the assistants that he ever hired: 'Is your goal to be a head football coach?' He always said if somebody had answered him, 'Not really, I'm OK just being a position coach,' then I don't think he really wanted him on his staff because he wanted ambitious guys.
Chuck Pagano is the fourth former Ravens defensive coordinator to land an NFL head coaching job.
"I think if you ask anybody they'd say yeah. That would be something you always work for and toward."
Ravens linebacker Paul Kruger believes Pagano has what it takes to be a head coach in the NFL.
"Chuck has a leadership quality about him. He's humble but he also knows when to take the reins and take charge," Kruger said. "He doesn't try to dominate you in every meeting. He's just a coach that knows exactly how players are and what direction they need. He's a hell of a coach and I really think he'll be a head coach one day."
One of Pagano's and Grigson's most important decisions will be helping Irsay determine Manning's future with the club.
In an interview with The Indianapolis Star this week, Manning appeared turned off by the sudden upheaval in the Colts complex.
"I'm not in a very good place for healing, let's say that," he told the newspaper. "It's not a real good environment down there right now, to say the least. Everybody's walking around on eggshells. I don't recognize our building right now. There's such complete and total change."
Manning said he met with Grigson last week and was told that Irsay would determine whether the quarterback will remain a Colt.
Manning said a meeting with Irsay "is going to happen at some point, but we haven't had that conversation yet because we really don't need to have that conversation yet."
He reiterated to The Star that he wants to remain with the Colts but realizes it's not his call solely to make. He also made it clear that he isn't contemplating retirement right now.
Manning is due a $28 million bonus in early March.
Information from The Associated Press contributed to this report.