Conan Byrne cancelled out Pavel Kamolov's opener but the Lithuanian side regained the lead through Kamil Bilinski with 20 minutes remaining.
But Gerard O'Brien struck three minutes from time to put the Irish side in a good position to qualify for the next stage.
Fellow League of Ireland side Drogheda also earned an impressive result, holding Swedish side Malmo to a goalless draw at the Tallaght Stadium in Dublin.
Northern Ireland's Crusaders took the lead against another Scandinavian outfit with a strong European pedigree, Rosenborg, but eventually lost 2-1.
Jordan Owens gave the home side the lead but John Chibuike equalised in first-half stoppage time and Jonas Svensson's 77th-minute goal made the Norwegians strong favourites going into next week's return leg.
Crusaders' national rivals Glentoran drew 0-0 away to KR in Reykjavik.
Welsh side Airbus UK drew 1-1 with Ventspils at Wrexham's Racecourse Ground, Chris Budrys equalising 10 minutes from time to cancel out Simonas Paulius' opener for the Latvians.
Another Welsh-Latvian clash in Rhyl ended with Prestatyn beaten 2-1 by Liepajas Metalurgs.
Jurgis Kalns and Ilja Sadcins gave Metalurgs victory, Andy Parkinson with Prestatyn's goal just before half-time.