Capt Arjun linked to this article, which in turn referred to an article in this month's Wargames Illustrated bemoaning the lack of historicity in "historical wargaming" today. The Capt's point (which I agree with) is that the spending of us wargamers (the "free market") is the key factor in shaping our hobby.
Reading through the article in Wargames Illustrated, the most interesting thing for me however was how the writer has divided rulesets into three groups, based on his personal preferences -
"Liked" (Tactica Ancients, WAB, WRG 6th Edition, to name the first few)
"Not liked" (WRG 7th edition, Vis Bellica, Fire and Fury, and DBM, amongst others)
"Hated" (DBA, Tactica Medieval, DBR)
The point I would make (on the understanding that rules preferences are intensely personal) is that gameplay suceeds or fails on the spirit that players take to the rules. Yes, a bad set of rules can encourage bad play, and yes, a faulty set of rules (contradictions, clearly malfunctioning mechanics etc) will definitely wreck a game, no matter how good the players are. However, the vast majority of mainstream rulesets are not broken or faulty.
I think of them as falling into the category of "can be made to work" - or to fail, depending on the spirit players choose to take the rules in. Once again, personal preference matters: if a ruleset turns you off, for whatever reason, you're less likely to be charitable with its shortcomings. Conversely, if a ruleset appeals to you (again for whatever reason - you like the historical period, or you've always wanted a Roman/VSF/Pulp/Starship Troopers/[fill in the blank] army) you're likely to give it the benefit of the doubt.
If it's the success or failure of gameplay we're talking about (rather than an assessment of the perfection of the ruleset) then it's as much the players as the rules that matter. Looking at the list that the writer of the Wargames Illustrated article (who goes by the pseudonym "D.I.S. Gusted, Lt Col (retd)") has made, the first thing I would say was that most of those rulesets can be made to work - the challenge is perhaps finding the right players for them. It requires that players be charitable to the rules, and focus instead on the game.
On the issue of historicity, for example, the writer already admits that a game need not be based on history to be "historical", giving an example of a game based on 70s TV police shows. It's the effort put in by the rules writers to evoke the atmosphere and spirit of the era (even if it is a fictional era, as with sci-fi or fantasy rules) - and I would add to that the effort made by the players to keep within the internal logic of that era.
After all, how do you stop un-historical play? If you knew that ancient Greek phalanxes never (to our knowledge) employed sweeping manouveres on the battlefield, you can either (a) write the rules to make it impossible for players to do so (b) write the rules to severely penalise players who do so or (c) hope that the players don't. (a) and (b) are only part of the solution: it's hard to write rules that prohibit actions without overly hampering players' freedom of play, and which penalise the majority of players for a minority's potential actions (in other words, (a) is a large hammer for a small nail). (b) is retroactive: the offending player may loose the game, but the game is lost for both sides anyway - there's not much fun to winning a game if your opponent incurs the wrath of the rules and accumulates a load of minuses by doing something ahistorical. It takes the fun out of it.
(c) seems overly optimstic at first glace - depend on the goodwill of players? But there is something we can do to encourage (though we can never guarantee) reasonable and historical play - and that is to make it easier for players to sense the historical period. In fantasy wargames, we call this fluff: background, maps, stories, pictures and illustrations etc, that help establish a sense of mood and environment.
Ultimately, this is also why I prefer historical wargaming to fantasy. I find the fluff more compelling and believable (it's hard not to: it has the advantage of actually having happened), and I can get into it more.