Olivier Giroud broke the deadlock for the Gunners with a fabulous finish in the 16th minute, only for Ashley Barnes to draw Brighton level before half-time.
Another fine strike by Giroud was cancelled out by Jose Ulloa as Brighton again hit back to make it 2-2, but Walcott's deflected strike five minutes from time took the Gunners through to the fifth round.
Brighton, hoping to add to their third-round scalp of Newcastle, started brightly with Will Buckley making some headway down the right channel.
A floated ball in from the left over a static Arsenal defence saw Barnes clear on goal with no offside flag, but Wojciech Szczesny was quickly off his line to narrow the angle and make a superb reaction save.
It proved decisive as Arsenal went up the other end and opened the scoring. As he had been against West Ham, Lukas Podolski was again the provider, creating space on the edge of the penalty area before feeding Giroud, who curled the ball into the bottom left corner for a third goal of the week.
Brighton, though, refused to lie down and were level on 33 minutes. A short corner was whipped in from the right, and Barnes charged through a sea of red shirts to get to the ball before Szczesny and power a header into the back of the net.
At the start of the second half, Arsenal full-back Carl Jenkinson got in on the overlap down the right after a surging run by Tomas Rosicky, but his snapshot was beaten away by Brighton goalkeeper Casper Ankergren.
Arsenal were back in front on 56 minutes with another brilliant finish from Giroud. Abou Diaby floated an angled pass into the left side of the penalty area, where the Frenchman held off Adam El-Abd, took the sting out of the ball with a lovely first touch and then lashed it into the net from six yards.
Brighton, though, hit back again and were back on level terms after 62 minutes. Arsenal again failed to deal with a cross from the right as Barnes' ball in was dispatched past Szczesny with a diving header by Ulloa.
With 20 minutes left, Wenger sent on Walcott and Jack Wilshere, replacing Rosicky and Alex Oxlade-Chamberlain.
An acrobatic effort from Giroud was then put behind, and from the resulting corner Walcott won it when his shot was deflected off El-Abd and into the net to send the Gunners through to Sunday's fifth-round draw.