Nate Silver over at fivethirtyeight.com just dropped a grenade into the wonkosphere. If you aren't familiar with fivethirtyeight, it's where you go if you want to know how good polling numbers are and the best publicly available insight into what those numbers mean.
One of the things I learned while exploring the statistical proprieties of the Iranian election, the results of which were probably forged, is that human beings are really bad at randomization. Tell a human to come up with a set of random numbers, and they will be surprisingly inept at trying to do so. Most humans, for instance, when asked to flip an imaginary coin and record the results, will succumb to the Gambler's Fallacy and be more likely to record a toss of 'tails' if the last couple of tosses had been heads, or vice versa. This feels right to most of us -- but it isn't. We're actually introducing patterns into what is supposed to be random noise.
Sometimes, as is the case with certain applications of Benford's Law, this characteristic can be used as a fraud-detection mechanism. If, for example, one of your less-trustworthy employees is submitting a series of receipts, and an unusually high number end with the trailing digit '7' ($27, $107, $297, etc.), there is a decent chance that he is falsifying his expenses. The IRS uses techniques like this to detect tax fraud.
Yesterday, I posed several pointed questions to David E. Johnson, the founder of Strategic Vision, LLC, an Atlanta-based PR firm which also occasionally releases political polls. One of the questions, in light of Strategic Vision LLC's repeated failure to disclose even basic details about its polling methodology, is whether the firm is in fact conducting polling at all, or rather, is creating fake but plausible-looking results in order to increase traffic and attention to its core business as a PR and literary firm.
I posed that question largely as a hypothetical yesterday. But today, I pose it much more literally. Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.
The discussions on his threads tend to be both sprightly and incomprehensible - with perl code and statistical expressions used without regard to the safety of bystanders.
If you want to see what "getting your ass peer-reviewed all over the bar" looks like, go watch. Nate seems to be holding his own, but it's kinda like watching fencing without knowing the rules, which are statistics and math.
But there are a number of other questions regarding the credibility of Strategic Vision, LLC, (http://www.strategicvision.biz ) one of which being that it is NOT the polling firm, Strategic Vision, Inc, (http//www.strategicvision.com)
So, not only confusingly similar names, but similar domain names. Similar enough to be grounds for legal action, I would think.
I'd consider that, coupled by the low-rent, post-office-box "Offices" to be a classic indicator of a possibly questionable group, as is the immediate reaction of Strategic Vision, LLC to earlier, more basic questions by Nate. Parsing this requires no understanding of the math or the science. It just requires an appreciation of human nature.
...in regards to Nate Silver's statements, we categorically deny them and will refute them. We have a call into our attorney on this and fully intend to take action that will vindicate us. I wish Nate had contacted me directly yesterday when he began this tirade, I could have answered his questions fully to his satisfaction prior to damage being done to our reputation. Now that he has made these accusations and posted them online, I must and will defend our company's reputation through all legal avenues available. The reason that we are going the legal route is he has attempted to do severe damage to our reputation and what is he going to do when we disprove him just say I am sorry. That isn't enough at this point.
The issue that concerned Nate Silver was that SV(LLC) does not disclose their methodology - how they collect their data from whom they collect their data, by what means, and what do they do with the data once they get it in order to wring meaning from it. Actually, he was more blunt than that, asking the most fundamental questions that one should never HAVE to ask firms of this type:
1. Are you actually polling anyone at all? Or are you just throwing some numbers up on a webpage and hoping nobody calls you on it?"We have a call into our attorney " is really all the answer a conscious human being needs, as
2. What is the location of your "offices" in Tallahassee, Madison and Dallas? Why is there no street address or phone number listed in association with them? How come none of the locations show up in a Yahoo! or Google search?
3. Why would you pick the name "Strategic Vision, LLC" for your company when the name "Strategic Vision, Inc." was already in use by an extremely well regarded, San Diego-based research firm that has been in business for more than 30 years? Are you deliberately trying to confuse your potential clients and leverage Strategic Vision, Inc.'s much stronger brand name?
a commenter on the tampabay.com story cited below points out.
Which leaves everyone wondering - what value can be placed on the SV(LLC) polls, which tend to break in favor of Republicans when SV(LLC) polls people about politics. It should be observed that many polling firms DO have a "house effect" that breaks one way or another, and that their polls are nonetheless considered entirely respectable. These firms generally ARE transparent about their data collection and data analysis. But there's a difference between a "house effect" and "lying," and if the question can arise, it must be resolved, so that nobody gets confused as to the difference.
The question is fairly important to Nate, considering that's what he does - evaluate polls. It's also a matter of concern to the polling industry as a whole, and that's how this whole mess arose. Jim Galloway explains:
[T]he American Association for Public Opinion Research, a kind of brotherhood of pollsters, publicly censured Strategic Vision for its failure to cooperate in an investigation into voter surveys conducted during the 2008 presidential primaries.The important part of the story for me, is this:
New Hampshire was the main focus – you’ll remember that Democrat Hillary Clinton surprised everyone with her win there. Twenty-one organization were asked to provide their statistical internals in order to help figure out what went wrong.
A final report was published in April. One finding: Techniques for developing polling models – weighting and such – are becoming increasingly murky. Often they’re declared to be trade secrets, and protected as such. The AAPOR argues that transparency is the only way to maintain public confidence in polling.
Said the association today, in a statement posted on its Web site:
Strategic Vision LLC was the only polling firm that did not provide sufficient methodological information (as defined by the AAPOR Code) about its surveys and refused to provide that information in response to AAPOR’s repeated direct requests.
For more than one year, AAPOR was unable to obtain the following basic information about Strategic Vision LLC’s polling in New Hampshire and Wisconsin: who sponsored the survey; who conducted it; a description of the underlying sampling frame; an accounting of how “likely voters” were identified and selected; response rates; and a description of any weighting or estimating procedures used. AAPOR considers the release of this information for public polls to be a minimum requirement for professional behavior among those who conduct public opinion research.
“What we are asking for is people in the profession to behave in a professional manner, and to release the methods through which they do their research – because these are crucial to understanding it,” Miller said. [Peter Miller, president of the AAPOR, is a professor associated with the Institute for Policy Research at Northwestern University.] “That’s our whole interest. That’s all we care about. We’re not trying to make some claim about the quality of Johnson’s research or anybody else’s.”
Here’s the catch:
“We’re saying we can’t know anything about quality if we don’t know what you did,” Miller said.
The argument in the thread at fivethirtyeight has raised the question as to whether Nate's analysis of SV(LLC's) polls (he used data from over a hundred polls they had done) has succeeded in pointing out data manipulation, or if it has simply detected noise that would ordinarily result from their sampling and data analysis.
I have no idea. At this point, I doubt anyone else does.
But then - as the The American Association for Public Opinion Research pointed out - that's precisely the problem.