I would guess it was Season 1. In this season, he brings his son along to audition for Mr. Finch the shoe salesman.
By Season 3 he is participating in a lonely hearts club (Floyd the Gay Deceiver) and in "The Bookie Barber" in Season 2, he is trying to move in on the imagined women friends of the bookie barber.
When Deputy Fife was mustering a posse in 'Quiet Sam' to rescue Andy, he was talking with Floyd on the phone. He mentioned bringing a gun, which Floyd panicked at, but said his wife had a big stick she used to prop the door open when she brought preserves up from the basement.
And now, I gotta chase them bothersome brats outta my barn so's I can make more moonshine.
Jubal, Jubal, Jubal