Town reached the Fourth Round of the Vase for the first time in 38 years after a rollercoaster penalty shoot-out at Walshes Meadow.
Rugby found themselves behind with only a minute on the clock, but a David Kolodynski double then gave them a deserved lead before the hosts levelled things up again to take the tie to its dramatic conclusion from the spot.
Danico Johnson and Kolodynski returned to Valley's starting line-up in place of the suspended Dylan Parker and the cup-tied Adam Shaw, with Jamal Clarke also returning after a spell out injured to replace Melu Mpande from the victory a week previously at Oadby.
The visitors almost made an unbelievable beginning when Caine Elliott surged at the heart of the Swifts defence straight from the kick-off before firing his shot over the bar from twenty yards.
However almost immediately after this, Town were themselves left stunned when 45 year old veteran striker Lee Hughes cleverly chipped the ball in behind the defence to allow Edilson Antonio a run in behind.
Ash Bodycote was quickly off his line to attempt to block the striker, but Antonio's tumble to the ground saw him rewarded with a penalty despite the protestations of the Rugby keeper.
Drew Canavan duly stepped up to stroke home from twelve yards to hand Valley the worst possible start to the game.
The early setback clearly knocked the stuffing out of Town for a while, and the one-time Premier League striker Hughes sent his effort high of the target as Stourport looked to take further advantage.
Rugby finally settled into the game though, and went on to dominate the remainder of the half.
Max Johnson's curled shot from the edge of the area was saved by home keeper Dominic Richards, before Town drew level on sixteen minutes when Kolodynski span neatly to fire home after Edwin Ahenkorah's initial strike had been saved too.
Valley soon looked to make further inroads on the slippery surface, with Kolodynski flashing in one particularly dangerous cross and then Max Johnson sending one over the bar.
A triple chance of taking lead went begging for Town, when Ahenkorah's on-target effort was blocked again, with Kolodynski's acrobatic overhead kick to the rebound smashing against the bar and Clarke heading the next follow-up into Richards's arms.
Ahenkorah had a further stabbed effort to Max Johnson's near post cross deflected away for a corner and Kolodynski spurned a great opportunity by steering the ball wide of the goal from a central position, as Town's superiority continued.
Luke English headed over the bar and Richards denied Caine Elliott with his legs, as Valley's frustrations of the first period continued up to its conclusion.
The visitors made a great start to the new half though, when English headed the ball back across the face of the goal to allow Kolodynski a straightforward finish with his head from close range.
However parity was restored within ten minutes, when the unmarked Edilson Antonio crept in at the far post to score from Canavan's floated free-kick.
A scrappy period of play followed this equaliser, but Town still managed to go close on a number of occasions within it - with Richards's reflex save denying Kolodynski, the busy Stourport keeper also diving across his goal to palm away Max Johnson's free-kick from the edge of the box and Ahenkorah twice prodding the ball wide from dangerous positions.
The Swifts then went onto to have their best spell of the game, with substitute Matt Gardner seeing his goalbound effort blocked on the line by English, before it was ultimately cleared by Clarke after a resulting scramble in the six yard box.
Valley's hopes of a late resurgence were then dealt a huge blow when the increasingly home side leaning referee sinbinned Max Johnson for dissent on 78 minutes to ensure he missed the majority of the remainder of the game, and the match official followed this up shortly after with a red card for Town boss Carl Adams for the same offence.
Rugby coped reasonably well with their numerical disadvantage though, and came the closest of the two sides to snatching a late winner when Clarke somehow lifted the ball over the ball from only a few yards out following a James Hancocks' free-kick.
And so it was to a penalty shoot-out that twisted and turned in both teams' favour at times. Loyiso Recci, Hancocks and Ahenkorah all successfully converted their spot-kicks to give Town a 3-0 lead, after Gardner had smashed his against the crossbar and Canavan was unable to repeat his success in normal time - thanks to a good low stop by Bodycote.
Jack Downing finally hit the target for the hosts, but it meant that Rugby only needed to score one of their final two of the initial phase of the shoot-out.
Unfortunately English blazed his effort way over the bar and new signing Dan Summerfield saw his strike saved by Richards, and with both Harry Higginson and Ben Tilbury both converting for Stourport, the tie went into sudden death penalties.
Max Johnson confidently stroked his home, but this was matched by Mason Birch.
Kolodynski's stuttering run-up led to another Richards' stop, but Bodycote prevented Antonio sealing matters with another excellent low save.
Barry Fitzharris scored with arguably the best penalty of the afternoon, to allow Bodycote to assume the position of match winner with his third save of the shoot-out to deny Joe Hanson.
Bodycote was subsequently submerged by the rest of the delighted Town squad and staff, and Valley were into the draw for the last 32 of the competition that they famously had won previously in 1983.
Valley Line-up: Bodycote; English, Hancocks, Loyiso Recci, Clarke;Fitzharris, Elliott, Max Johnson; Ahenkorah, Danico Johnson (Summerfield), Kolodynski. Subs not used: Omar Recci, Marsden, Thomas. Booked: Loyiso Recci. Booked (Sin Bin): Max Johnson.
Stourport line-up: Richards, Hanson, Jackson, Craddock, Higginson, Tilbury, Canavan, Birch, Hughes, Antonio, Townsend. Subs: Sinclair, Allchurch, Downing, Danks,Bishop, Gardner.