Brandon Jennings had 10 assists but made just three of 15 shot attempts and scored six points.
Ellis cut it to 94-92 on a jumper with 1:50 left, but Toronto's Kyle Lowry replied with a 3-pointer. Lowry scored 15 and was one of six Raptors to reach double figures.
The Hornets' Frank Kaminsky will undergo a procedure to remove a small pocket of air between his lung and chest wall and will need approximately six weeks to recover.
Cavaliers general manager David Griffin says the team's best basketball is in front of them and that he intends to keep this group together.
The latest notes, updates and analysis on the Magic from around the web.