Common misconception: "Vinyl LP and 30 IPS tape are inherently superior to Compact Disc Digital Audio because they're analog."
Fact: The human ear can hear frequencies from about 20 Hz to 20 kHz because of the band-pass features of the various parts of the ear. By the Nyquist-Shannon sampling rate theorem, 44 kHz sampling can perfectly reproduce any signal from DC to 22 kHz.
A 96 kHz sample sounds better not because the source contains frequencies in the 22–48 kHz range but because it's easier to make analog low-pass filters that transition from 20 kHz to 48 kHz than 20 kHz to 22 kHz. Any oversampling DAC will produce the same gains.
More bits per sample means less noise added to a signal. But the ear also can't hear more than 20 bits in practice because it can't hear anything below 0 dB sound pressure level (SPL). The THX standard specifies a 75 dB SPL for a -30 dB reference signal (that is, 105 dB SPL for a rail to rail signal). A 20-bit linear PCM encoding gives a -120 dB quantization noise floor, meaning that it can reproduce sounds as quiet as -15 dB SPL. Most CD music is mastered for a -12 to -6 dB reference signal (thus reproducing a rail to rail signal at 87 dB SPL); the -96 dB noise floor of the 16-bit linear PCM used on CDs becomes 9 dB below audibility. Even if we get into a whisper-quiet passage played at 30 to 35 dB SPL, and 16-bit linear PCM begins to use only the region around +/- 127, the ear still can't hear the quantization noise because it's below 0 dB SPL. Moreover, modern CD mastering uses noise-shaped dither patterns that shift dither noise above 16 kHz range, where the ear can't reliably hear even 30 dB SPL.
When you refer to problems with CDs, make sure you're not conflating the format with the music. Many audio CDs that seem to lack punch sound that way because a loudness war has been going on since the late 1990s, with each label trying to make a "hotter" mix than competitors. CDs ship with an overcompressed mix that even occasionally clips, taking the kick out of the kick drum on the best equipment. Californication by Red Hot Chili Peppers ends up nearly unlistenable due to this. The excuse used to be that they're mastered for radio play because people in motor vehicles typically prefer music at a constant level above engine noise, and the dynamics need to be compressed to fit. But radio stations have their own equipment to compress for FM radio, which is different from compressing for CD because of the preemphasis inherent in FM transmission.
Dolby Digital audio samples at 48 kHz, 24-bit, compressed down to 384 kbps, and that's more than enough for transparent 5.1 motion picture audio reproduction. An Ogg Vorbis encoding at about 192 kbps reproduces stereo audio flawlessly to even an above-average human ear. Or are you shopping for music for your dog?