The South Florida native was optioned to Triple-A Tucson following Thursday's 5-3 win over the Marlins.
After tearing up the Pacific Coast League, Rizzo joined the Padres on June 9 to make his major league debut. But he batted only .143 with 36 strikeouts in 98 at-bats.
"He handled himself great for a 21-year-old coming up to the big leagues for the first time," manager Bud Black said. "There was a lot of hype, and he handled it well, even through adversity. It's time for him to go back and get back on track and get his confidence back."
First baseman-outfielder Kyle Blanks was recalled from Tucson and will join the Padres on Friday.
Rizzo had a big cheering section of family and friends this week for the Padres' sweep of the Marlins. But his hitless streak grew to 13 at-bats, and he conceded his confidence had been shaken.
"I've just got to get my swing back," he said. "I believe I'll be up here for a long time in the future. Obviously I was struggling and wasn't putting good swings on the ball. I've just got to go get that back."
Rizzo made 28 starts for San Diego, all at first base.