Valley exited the FA Vase competition in disappointing fashion at the hands of their lower level hosts following another below par away performance.
Dylan Parker had given Town the perfect start with the opening strike in the second minute, but they were unable to kick-on thereafter and a headed goal to a set-piece halfway through each half gave Dudley a deserved scalp.
Justin Marsden came in for Keenah Rosser in the only change to Rugby's starting line-up from the previous weekend's outing at Oadby, and Town were soon ahead when Parker was the quickest to react to Sam Lomax's long punt over the top of the Sports' defence to apply a neat finish past Lomax's opposite number Declan Haire.
Dudley responded well to the early setback, with Valley skipper Loyiso Recci twice denying Nicholas Turton with well-timed tackles and Mitchell Evans then testing Lomax with a low shot.
Parker nearly slid onto Lee Thomas's cross to the near post, but it was the home side that were generally in the ascendance and their reward came when David Meese powerfully headed home a deep free-kick into the box.
A great opportunity for Town to restore their lead went begging shortly after though, with Parker again in behind the Dudley defence - but this time he chose to square the ball to strike partner Danico Johnson, who missed the gaping target with a bobbled effort.
Parker sent a volley wide of the frame after neat control on the edge of the penalty area, with the same player also denied by a great tackle from Kyle Pearson as the half-time whistle approached.
Rugby boss Carl Adams made two changes for the start of the second period, with Rosser and Jamal Clarke on for Justin Marsden and Ryan Seal, but it was more of the same stuttering fayre from his side.
Johnson's low free-kick from a relatively dangerous position took a deflection off the wall and was clumsily gathered by Haire, but the Dudley stopper was in more convincing form when he dived full-length to keep out Recci's header to another Johnson set-piece delivery.
The hosts were far from out of it though, and the winning goal originated from sloppy defensive play by Caine Elliott when the youngster was dispossessed by James Weaver - who ran on to draw a decent save from Lomax.
From Mitchell Evans's resulting corner, Dudley defender Matthew Vaughan's run was left unchecked and he looped his header into the net on 64 minutes.
Town huffed and puffed thereafter as they looked to take the tie to a penalty shoot-out, and they did create a few chances to draw things level.
Haire was again at full stretch to keep out Johnson's shot to Clarke's lay-off in the box, and Charlie Evans saw his effort blocked on the line, whilst at the other Lomax kept his side in it by tipping over a rising strike from Matthew Hearsey.
A couple of late scrambles also failed to go Rugby's way - the last of which seeing Recci poke one agonisingly wide in the dying minutes.
Line-up: Lomax; Elliott, Recci, Francis; Evans, L Thomas, Hancocks, Marsden (Rosser), Seal (Clarke); Parker, Johnson. Subs not used: J Thomas, Padmore. Booked: Evans, Hancocks.
Dudley line-up: Haire, Roberts, Pearson, Evans, Vaughan, Hayer, Weaver, Meese, Turton, Clarke, Hearsey. Subs: Conway, Westwood, Bhandal, Latham, McDonagh.