The Black Cats find themselves bottom of the Premier League having claimed just one point from their first four games.
On Saturday, they travel to the Hawthorns to face 19th-placed West Brom, with Albion having won the last three meetings between the two clubs.
However, Gardner, who scored from the penalty spot as the Black Cats went down 3-1 at home to Arsenal last weekend, has called for calm on Wearside.
"I don't think we should be too worried about it yet," he said. "It is too early to be getting into the league table too much.
"The league is tight - one win, and you're back in the top half or whatever. You can't really worry about it too much at the moment.
"It was the same last year. 12 or 13 teams were in the middle, and there was hardly anything between them. As long as we keep putting performances in like we did in the second half against Arsenal, we will pick the points up."
Di Canio defended
On Tuesday, Sunderland boss Di Canio accepted an FA misconduct charge and was fined £8,000 having been sent to the stands on Saturday for arguing with referee Martin Atkinson.
The Italian has won just three of the 12 league games he has presided over since replacing Martin O'Neill in late March, resulting in criticism of his approach.
But Gardner insists the former Swindon boss is still trying to change the mentality on Wearside and should not be judged so harshly.
"I think a lot of the people who've been writing the manager off have got something against Paolo Di Canio," he added.
"It's not just his management skills, there's always something to knock him about. Of course he's a Premier League manager - he's managing in the Premier League, and he's trying to change things here.
"We've still got faith in him as a squad, of course we have. The lads want to go out there and put in as much hard work as we do on the training pitch and go and get that first win that we want."