John Toshack's men could not stretch their run of two successive victories to a third despite a spirited display by a young Wales side.
They conceded two goals late in the first-half, one a debatable penalty converted by the best player on the pitch, Stevan Jovetic, and the second a fine shot on the turn seconds before the break from Radomir Dalovic.
That was a killer blow for Wales who thought they had got to the interval still in contention.
Vokes came on as a second-half substitute and scored with a cleverly flighted header, but despite sterling efforts from the Welsh they could not fashion another strike.
Boss Toshack made four changes from the side that performed so well to win in Azerbaijan in June.
Midfielder David Edwards was out with a groin strain, but Toshack was able to recall three fit-again defenders to bolster his side in the Gradski stadium.
James Collins and Sam Ricketts returned, along with Danny Gabbidon, the West Ham defender, for his first Wales game since November 2007.
With Craig Bellamy withdrawn by Manchester City, young Reading striker Simon Church partnered Robert Earnshaw up front.
Earnshaw had the ball in the net inside the first minute, firing wide of Vukasin Poleksic, but the Nottingham Forest striker was penalised by referee Milorad Mazic for a push on Miodrag Dzudovic as he burst through.
Montenegro possessed clever, quick midfielders, and they caused Wales plenty of problems, the excellent Stevan Jovetic involved in everything going forward for the home side.
But Wales created chances of their own. Joe Ledley combined with Aaron Ramsey, the Arsenal youngster, and then chipped the ball over the defence for Church to instantly volley just wide of the far post.
Earnshaw had a shot that looked to have been handled in the area, and Collins saw a snap shot from 18 yards held by Poleksic.
Wales were getting their passing game going, and when Ledley and Ramsey combined neatly down the right, the captain fired in a shot on the turn that flew inches wide.
Jovetic, the Fiorentina forward, was a constant threat and created space for himself on the edge of the area to test Wayne Hennessey, and a minute later Jovetic got away down the left and lifted in a cross that Radomir Dalovic headed just over.
Montenegro took the lead after 31 minutes, the Serbian referee awarding a soft penalty when Simon Vukcevic went down under a challenge from Gabbidon.
Vukcevic, from Sporting Lisbon, looked to be backing in and on his way down before any contact from Gabbidon, but Jovetic was given the chance to fire the penalty past Hennessey and did just that.
Six minutes later Earnshaw sprinted clear of his marker, but his final touch was poor and it delayed his shot long enough for Poleksic to save at the second attempt.
But in first-half injury time Wales were caught out on the right when Vukcevic's pass found Dalovic and Rapid Bucharest striker squeezed a cross-shot inside Hennessey's far post for the second.
Wales made three substitutions at break, sending on goalkeeper Lewis Price for Hennessey, Lewin Nyatanga for Williams and Sam Vokes for Church. Montenegro also replaced their keeper, Mladen Bozovic coming on for Poleksic.
And Wales hit back after 52 minutes. Ricketts launched a deep cross in from the left and Vokes got above his marker to glance a header down and just inside the post, his second goal for his country.
Five minutes later, Blackpool's new signing Neal Eardley took over from Ricketts, with Chris Gunter switching from right to left back in the process.
Montenegro sent on Mladen Kascelan for Pekovic after 62 minutes, with Wales replacing Collins with Peterborough's Craig Morgan.
But Jovetic was still a danger. He got away on the left and saw a curling effort crack come back off the far post, with Vukcevic missing the rebound.
Miodrag Dzudovic was booked for a bad foul on Earnshaw, before Lika Pejpovic replaced Radoslav Batak after 68 minutes.
The host sent on Nemanja Nikolic and Fatos Beciraj for Bozovic and Dalovic, while Ramsey made way for David Cotterill.
Mitar Novakovic came on for Vukcevic with five minutes left, Montenegro heading towards victory.
Wales kept going to the end, but could not find their way through the home defence again.