Thinking about this a bit more, there are still some natty issues that need to be resolved. First, there is the definition of the population (sample space), which a lot of people are unhappy with. It seems as if a LOT more people read IW than subscribe to it (after all, it does get passed around departments a lot). That doesn't take into account the IE people either. I was thinking maybe allowing people to submit requests to be part of the population (non-subscribers) and then choosing from there, but then you might get (to use your phrase) a lot of fanatics who might not ordinarily be part of your reader base. Maybe registering on IE for a minimum of 6 months?
Second, the sample size problem doesn't go away (it never does). Ferinstance, a smaller sample is more influenced by outliers. Eg., say you have 50 Windows respondents but only 10 OS/2 respondents. If 49 of the Win guys rate the OS as an 8 (of 10), and one gives it a 2, you get an average rating of (49*8 + 1*2)/50 = 7.88.
But if nine OS/2 guys rate it as 8, and one gives it a 2 you get (9*8 + 1*2)/10 = 7.4. I assume whoever you hired to do this would weight those factors accordingly, but it pays to keep it in mind.
Of course, even with that problem, it's still better than a winner-take-all approach like it is now.
For questions or comments, email the Forums Editor