The France international only moved to St James' Park in January, with a sizeable fee required to secure his services.
He has, however, struggled to settle in England and has suffered a dip in form as a result.
His former club Montpellier have opened the door for a return in January, with chairman Laurent Nicollin of the opinion that a deal is there to be done.
Pardew, though, insists that Yanga-Mbiwa remains part of his plans and is confident he will prove his worth once he finds his feet in the Premier League.
On the transfer rumours, he said in The Independent: "Everybody has the right to an opinion, but that particular one is not an opinion I share.
"We paid a lot of money for him and you're looking for a bit of character in a player in this situation. That's what we've seen from him.
"If you take a defender out of the side, you have to say why and give them some solutions. I'd like to think I've done that for him and we need to get the kind of performance out of him that he's put in for this club in the past.
"It's very difficult for defenders to adjust to the Premier League, much more so than for attackers because I think the intensity catches them out.
"He's come out of the side, had a look at it and now he's back in again. I'd like to think he'll show his character."