There is a really good reason for this: when you whisper, you don’t voice anything (try it!). If you’re trying to feel/hear a voicing distinction, you will not be able to do so when whispering because you are not in fact producing a voicing distinction. (This is also why people with laryngitis, i.e. inflammation of the vocal cords, can only speak in a whisper.)
How, then, do we understand the difference between, say “buy” and “pie” when whispering? It’s probably partially other factors like aspiration and vowel length, and partly the way we distinguish between “pi” and “pie”: context.
This post brought to you by my former self as a 14-year-old proto-linguist having incredible difficulty distinguishing between eth and theta.