Why Nate Silver is Wrong

Famed pollster and sabermetrician Nate Silver is calling the US Presidential race for Obama, in a big way:

Silver’s mathematical model gives Obama an 85% chance of winning. The Presidential election is based on an electoral college system, so Silver’s model rightly looks at state-level polls. And in swing state polls, Obama is mostly winning:

This is slightly jarring, because in national polls, the two candidates are locked together:

So who’s right? Is the election on a knife-edge like the national polls suggest, or is Obama strongly likely to win as Silver’s model suggests?

While the election could easily go either way depending on turnout, I think Silver’s model is predicting the wrong result. In order for that to be the case, the state polling data has to be wrong.

There are a number of factors that lead me to believe that this is the case.

First, Republicans tend to outperform their poll numbers. In 2008, the national average got the national race just about right:

In the end, Obama won the election with 52.9% of the vote, against McCain who came out with 45.7%.

However, polls have historically underestimated Republican support. Except 2000 (when a November Surprise revelation of a George W. Bush drunk-driving charge pushed Gore 3.2% higher than the final round of polling), Republican Presidential candidates since 1992 have outperformed their final polls by a mean of 1.8 points. Such an outcome for Romney would put him 1.5% ahead in the national polls, and imperil Obama’s grip on the swing states.

Second, the Bradley Effect. The interesting thing about the swing states is that many of them are disproportionately white. The United States is 72% white, but Iowa is 89% white, Indiana is 81% white, Ohio is 81% white, Minnesota is 83% white, Pennsylvania is 79% white, New Hampshire is 92% white, Maine is 94% white and Wisconsin is 83% white. This means that they are particularly susceptible to the Bradley Effect — where white voters tell a pollster they will vote for a black candidate, but in reality vote for a white alternative. In a state in which Obama holds a small lead in state-level polling, only a small Bradley Effect would be necessary to turn it red.

This effect may have already affected Barack Obama in the past — in the 2008 primaries, Obama was shown by the polls to be leading in New Hampshire, but in reality Hillary Clinton ran out the winner. And many national polls in October 2008 showed Obama with much bigger leads than he really achieved at the polls — Gallup showed Obama as 11% ahead, Pew showed Obama as 16% ahead.

A small Bradley Effect will not hurt Obama where he is 7% or 11% or 16% ahead in the polls. But when polls are closer — as they mostly are in the swing states — it becomes more plausible than such an effect could change the course of the race.

And the Bradley Effect in 2012 may be bigger than in 2008. A recent poll by the Associated Press concluded:

A majority of Americans (51 percent) now hold “explicit anti-black attitudes” — up from 49 percent in 2008 — and 56 percent showed prejudice on an implicit racism test.

Finally, polls have tended to overestimate the popularity of incumbent Presidents, especially Democrats. In 1980, polls put Jimmy Carter 3% of his final tally, and in 1996 polls put Bill Clinton 2.8% ahead of his final tally:

Taken together, these difficult-to-quantify factors pose a serious challenge to Silver’s model. While it is fine to build a predictive model on polling data, if the polling data fed into the model is skewed, then any predictions will be skewed. Garbage in, garbage out.

I rate Obama’s chance of being re-elected as no better than 50:50. If Silver really rates his chances as 85:15, perhaps he should consider taking bets at those odds.

UPDATE:

Obviously, Silver’s predictive model (and far, far more importantly the state-level polling data) proved even more accurate than 2008. However, the 2010 British General Election (in which polls and therefore Silver vastly overestimated the Liberal Democrat support level, leading to an electoral projection that was way off the mark) illustrates that there remain enough issues regarding the reliability of the polling data to ensure that Silver’s model (and similar) continue to suffer from the problem of fat tails. With solid, transparent and plentiful data (as Taleb puts it, in “Mediocristan”) such models work very, very well. But there remains plenty of scope (as Britain in 2010 illustrates) for polls to be systematically wrong (“Extremistan”). Given the likelihood that every news network will have its own state-level poll aggregator and Nate Silver soundalike on-hand come 2016, that might well be a poetic date for the chaotic effects of unreliable polling data to reappear. In the meantime, I congratulate the pollsters for providing Silver with the data necessary to make accurate projections.

About these ads

67 thoughts on “Why Nate Silver is Wrong

  1. Silver is not doing some blind averaging of the polls. John thinks that the unique aspect of Silver’s method is that it disaggregates the national data to focus on the Electoral college, but that’s just one aspect.

    Silver understand that not all polls are created equal, and he accounts for the historical performance of each poll.

    Which is, when John writes “However, polls have historically underestimated Republican support.” he is not providing Silver with any new information.

    I’m not saying this because I’m shilling for Obama (or Romney). I’m saying that the reasons John gives for questioning Silver’s methods are not fully informed by an understanding of the method itself.

    I think Silver has it right.

    • Which is, when John writes “However, polls have historically underestimated Republican support.” he is not providing Silver with any new information.

      I know I am not providing Silver with any new information. I am just disagreeing with how he is interpreting the information we both have access to. Silver seems to believe that Bush’s fall in 2000 from the last poll data (caused, as far as I interpret it, by the drunk driving revelations) and Obama in 2008 (when the polls got it roughly right) suggests that this effect is less significant than I do.

      I know Silver’s method, and it worked fantastically well in 2008. But this is a unique race and there are substantial reasons to believe that his parameters are flawed this year. The Bradley Effect is likely to play a big role.

      • Then to boil it down, you think Silver has it wrong because (according the the AP article), Obama’s blackness is not accurately represented in Silver’s understanding of systemic bias in poll data. Moreover, the outcome of the 2012 election is more sensitive to this error (because it is closer) AND the effect will be larger than in 2008.

        But the evidence supporting this logic is circular. The AP conclusions are based upon a poll.

        Why would Americans admit racism to the AP, but not to a pollster asking about whom they intend to vote for?

        The fact that more Americans in 2012 admit to racial bias might be an indication that the Bradley Effect is fading, if it means that admitting racism to a pollster is no longer stigmatized.

        • Well, it’s not entirely to do with blackness — that is just one of a number of factors. I think the phenomenon is more similar to something I am very familiar with in the UK known as a “Shy Tory Effect”, whereby the Conservative Party almost always under-performs its numbers. People don’t want to admit over the telephone to be voting for a political party that is widely seen in much of the media as “selfish”. Obama’s blackness, I hypothesise, is one of a number of factors leading to an even stronger form of this — people in telephone polls not wanting to admit to voting for a party (the Republicans) widely seen and portrayed as not only selfish, but also racist.

          That may be the difference between the AP poll and telephone polls for the Presidency — people in an anonymous poll online may admit their prejudices more readily than over the phone. (Online polling in the UK in some cases successfully cancelled the “Shy Tory Effect”).

          It will be impossible to say whether I am right or Silver is right after Tuesday, because we will only have one result. A resounding win for Obama is certainly possible, and will look good for Silver. So too is a marginal win for Romney, which would look good for me. But because we cannot run the election again and again and do a regression, “right” and “wrong” are very subjective terms in this debate. But if I was Silver, I would find the contrast between 2008 — where he was 11 points ahead in the Gallup poll, and won by 7 — and now where he is 5 points behind alarming. Putting a candidate who is behind in some national polls as an 85% chance to win is a huge risk to one’s reputation. If Obama was polling at his 2008 numbers, an 85-90%+ estimate would be more justified.

  2. It will be impossible to say whether I am right or Silver is right after Tuesday, because we will only have one result.

    No, no! There will be many results.

    For example, Silver has extended his method to other elections (e.g., Senate). This will allow assessment of whether the method has accurately estimated the outcome probabilities.

    In fact, Silver will do exactly this, and recalibrate his model accordingly.

    • I consider the whole thing as one discrete event on the scale of a “Presidential contest” just as I consider 2008 to essentially be one discrete event.

      • i get that your post is just about the Presidential election, but Silver’s method is testable.

        Let’s say Obama wins, but the distribution of other races is way off what Silver predicted. Well, that reinforces your criticism that Silver is unable to extrapolate estimates of future probability from examination of prior analogs. In this case, he would just be lucky with the Presidential election.

        You criticism does stand. The future poses unique challenges, and probabilistic analysis of the past is limited in its ability to incorporate unique circumstances.

        But fortunes are lost arguing “This time is different.” The burden of proof is always on the exceptionalist.

  3. Seems to me that Obama has served his masters quite well and will be “rewarded” with four more years [although one must wonder exactly what kind of person would want such a job at this time].

    It’s like which would you rather be assaulted by, a hoodlum with a .38, or one with a .45?

  4. Pingback: Good post on why Nate Silver is wrong « Rhymes With Cars & Girls

  5. John: You do mention “turnout”, but I don’t see that you consider (speculation on) it in your arguments. Dick Morris, former Clinton campaign adviser and current FoxNC contributor, predicts big Romney win on shifts in turnout not recognized by polls.

    P.S. Who the hell is Nate Silver and what is a sabermetrician?? Are they “privy” to the UK?

    • Yeah, I think turnout is hard to predict. Basically I was considering earlier how it would be easier to predict the result of an election if everyone votes, but much harder if turnout is a variable. Right now it’s just another layer of chaos and complexity I’m trying to peer through. My sense (from polls that consistently show that self-described “conservatives” are by far the largest political affiliation in America) is that many who stayed home in 2008 from disillusionment with Bush will show up to try and throw out Obama in 2012.

      Nate Silver is a blogger who writes for the New York Times.

      Sabermetrics is a system for predicting baseball results. Very very un-British.

      • And, I daresay, very insensitive to human fickleness, ignorance, emotion, gullibility — all of the Seven Deadly Sins except, maybe, gluttony. [I am assuming “baseball results” is not about betting, sportswriter forecasts, etc. — only who scores the most runs and wins].

    • Clarification: “shifts” refer to different turnout levels in election years 2004, 2008, 2010 and estimated 2012. I should have added “sufficiently” as a modifier of (not) “recognized” (by pollsters).

  6. I think you’re misinterpreting the large gap between the 85% and the 15% chances Nate Silver gives to Obama and Romney respectively. Yes, that is an extremely large gap, but that does not necessarily mean he believes the difference in popular vote will be so large. Obama is leading in a majority of state polls, and given the short time til election day, Silver believes he’s very likely to win these states, even if by just a small margin. McCain only won Texas 56% to 44% in 2008, but that still counts as a 100% win.

    Take a look at his popular vote prediction: 50.6% to 48.3%. That makes the margin seem much less pronounced. Also, he actually did offer to make a bet with Joe Scarborough, but he was widely criticized for doing so.

  7. Take a look at his popular vote prediction: 50.6% to 48.3%. That makes the margin seem much less pronounced.

    Not really. I’m looking at the RCP poll of polls which has a much narrower margin: 0.2%. I’m looking at Gallup that has Romney 5 points up. I’m looking at all this in light of the well-observed Bradley Effect that occurred in 2008 (Gallup had Obama up 11, Pew had him up by 16, he won by 7). No, no way will Obama win by 2.3%. And even if Obama was up 2.3% in the polls (he’s not) I still wouldn’t say he had an 85% chance of winning. There are way, way, way too many layers of complexity — turnout, the fact Republican candidates tend to do better at the ballot box than the polls, etc, etc.

    Also, he actually did offer to make a bet with Joe Scarborough, but he was widely criticized for doing so.

    Yeah he offered a $1,000 bet at even money, an extremely poor bet given that you can get much better odds elsewhere. I’m talking about offering 85:15 (17:3).

  8. If I know the judgement of the fanatical USA voter, they will pick Romney, because he looks like a Hollywood Actor President.

    Its a popularity contest stupid.

  9. Is it possible that the fundamental differences in how data is collected for sports statistics and election polls mean that analysis in one domain is not particularly relavent to successful analysis in the other?

    Personally I enjoy looking at the statistical probability of particular professional sports teams winning and then watch the season with anticipation of how the various dynamics effect the outcome — players get injured, managers get fired, trades are made. The odds are readjusted before every game and the eventual championship odds are rarely lopsided. At the end of the season.

    Mr. Silver would like to beleive that he has a sophisticated “meta – model” that has predictive value with greater reliability than any indivual poll whose data is aggregated. I beleive that is a mathematical impossibility.

    I would posit that the situation is not dissimilar to the continued efforts of financial “chartists” that offer increasingly sophisticated meta-analysis of the movement of stock prices, futures and derivatives. Unfortunately though the time lag between using the essential backward looking explanatory model continues to decrease as computers become ever more powerful the fact remains that forward looking predicition is still not completely accurate. Many learned academics have failed rather spectacularly when they (and others) try to put theory into practice.

    The fact is the actual “proof of simulation” for any sports match-up happens with each game played, analysts can tweak their formula mutliple times per week.

    The same sort of “on the fly” adjustments can happen for financial models with every execution of a trade — wider ranges of “black swan” type events are actually a boon to those who trade in the less probable price shifts. A well known risk vs. reward gradient exists for things like interest rate moves over time and other such relavent (and hedgeable) indicators.

    Mr. Silver is on the second “proof of simulation” under economic, foriegn policy, and incubement circumstances that are wildly different than those of four years ago. If he is successful in predicting even 46 of the states I would strongly encourage him to immeadiately setup shop with a prominent hedge fund, make a huge amount of money very quickly and then retire. If he is not accurate for more than 45 states he will likely be able to learn much from the “second run” of his prediction and can reasonably be expected to garner enough interest from political scientists, econometrics researchers and the huge base of online and print news / opinion sites to profitably pursue both a quasi-academic and pundit-based career for at least another four years…

    • Is it possible that the fundamental differences in how data is collected for sports statistics and election polls mean that analysis in one domain is not particularly relavent to successful analysis in the other?

      Very likely.

  10. Having worked as a Pollster, in my youth (Ringing people to conduct a Poll) some people do feel it is an opportunity to vent their spleen, others are feeling judged by the Pollster, so I do think there is evidence of the “Bradley Effect”. Others are angry at being disturbed so they give incorrect information.

    But statistical sampling can’t account for people’s confidence of anonymity in the polling booth.

    Personally I think poll results should be banned in the media, as it influences people who don’t think, but go with whoever seems popular at the time.

  11. Well, one thing is for certain, Nate Silver gave you something to talk about. Something meaningless, but something.

    A vote for Obama is a vote to end Social Security.

    A vote for Romney is a vote for gun control.

    The lesser of two evils is still evil.

    I think the evil that is Obama wins.

  12. I think Nate Silver has it right – Obama will win resoundlngly. As to whether his polling is a useful predictor – as another commentator said – he’s made a lot of predictions in congressional races so his methods will be proven out or not.

    I’m not sure about the Bradely effect re Obama – he’s a known quantity – ppl either like him or hate him and won’t be shy to tell a pollster. Has a Bradely effect been measure with an incumbent black politician? That would be interesting to know.

  13. John, maybe you should stick to blogging about things that you know, instead of those you don’t, like mathematics and electoral politics, lest you look a fool.

    • Nate and the national polls put Obama 2.3% ahead of Romney in the popular vote, but it looks like Romney was very close in the popular vote in much the way I described he could be in the post — the swings I talked about from the Bradley Effect and from the Incumbent Effect seem to have materialised, at least to some degree, especially if Romney wins the popular vote.

      As it happened, the votes didn’t add up in the right places, and Obama has won the electoral college. Given that I gave him a 50% chance of winning, this tallies very much with what I wrote in this piece. There WAS a swing to Romney from where Silver projected, just as I stated there would be, it just was not big enough for Romney to win. A little more and it would have been enough, he could have grabbed Ohio and Florida, and I still think 85% two days ago was wildly out of whack. We’ll see tomorrow what the final outcome is in terms of the popular vote, but I think the qualifications I added to Silver’s forecast hold up very well.

      • Your critique of Silver isn’t holding up at all, because Silver is not the least bit concerned about the popular vote. E.g., if Silver were to issue an opinion on a cricket match, it would be pointless for you to critique his understanding of football.

        For you to claim that the 85% estimate was out of whack, you need more data.

        There are 50 states, plus the District of Columbia. That means that Silver made 51 estimates of probability distribution in the US Presidential election.

        Go ahead and look at which states he got wrong. If there really was a systematic error in Silver’s method, then you would expect that the states he predicted would be very very close would go disproportionately to Romney.

        This doesn’t even consider Silver’s predictions of the Senate races, which use the exact same method.

        Actually… you don’t have to do this. Nate Silver will do this for you. He will publish an analysis of his own results, thereby saving you the trouble.

        In the meantime, you should hold off on your victory speech.

        • I have no victory speech, but if Romney wins the popular vote or as he appears to have done at least comes very close then I will say that the national level phenomena I alluded to had a serious impact on the race, just as I suggested they would. I expect Silver to get most of his raw state predictions right, 47, 48 or 49 out of 50, or maybe even a full score but there was a lot of chaos and complexity bubbling under the surface that so very easily could have changed the result. 538 totally missed the result of the last British General Election, for example. Nate makes good predictions in Mediocristan, but the tails in electoral politics (Extremistan) are very fat.

        • Since I don’t bet with the pros, I may not understand “odds”, “chances’, etc. But I would say that any adviser predicting an 85% chance (.85 probability?) of an event that turns out this close would get fired. Isn’t this what it means? If 100 similar/identical events were held, 85 would result in a win by the Obama-case.

        • I guess that if we ran the last 10 days of this event 100 times, we would see 100 wildly different outcomes. Maybe more Obama wins than Romney wins, but I’d be very, very surprised if it was 85:15. 65:35 at most, is my speculation.

      • Were you watching the same election I was watching? California dumped millions of votes into Obama’s margin, and the election wasn’t anywhere near close at 1 AM, the time you posted your comment. I’d be embarrass ed to defend predictions as bad as yours when it was completely obvious that they weren’t going to come true.

        • What predictions? All I did was give my own probability distribution and give some possible reasons why it might to some extent differ from Silver’s. I never made any specific predictions, and nothing that occurred was inconsistent with my implicit model. Although earlier results showed Romney in the lead in the popular vote (all the way up to 75% returned!) it does turn out that the Bradley and Shy Tory Effects seem to have been minimal, and the state-level polling proved relatively accurate allowing Silver to make accurate projections. At no point did I rule any of that out or even say it was unlikely. Silver’s model wouldn’t be popular and well-known if it didn’t make good predictions most of the time. At no point did I say he was lucky in 2008 or 2010. He has a sound, rigorous model.

          What I did was identify a fragility in Silver’s model, specifically the unreliability of polling data which has historically been shown (as in the UK election of 2010) to skew Silver’s model, and gave an estimate of what I conceived as the real underlying probability taking into account that the state-level polling could be skewed. Essentially what I did was purchase a convex (low cost, high payoff) call option on Silver being off, which he has been in the past. It turns out my call option didn’t reach the strike price. Big deal. Black swans exist — the fragility I have identified is real, and Silver’s model will fail again at the tails in the future.

        • Who cares what the early returns were?

          If you don’t know enough about American electoral politics to realize that the West Coast was going to dump MILLIONS of votes into Obama’s margin, and that the West Coast has the last poll closings, maybe you shouldn’t make predictions and back them up with bullshit when in fact, you don’t know what you’re talking about.

          Stick to economics. Avoid bluster on subjects of which you are ignorant.

        • The issue is not the numbers (plenty of others thought Mitt could win the popular vote even when Obama had won the electoral college) but the hard-to-quantify tail risk of the polls being wrong. It has already happened once in recent years (2010). It could have easily happened this year on turnout because of the definition of a “likely voter” (among other things). As it happened, the polls nailed it. Great. I hope we can learn from what went right this year, because sooner or later there will be a year where we get it very, very wrong.

      • 538 totally missed the result of the last British General Election, for example.

        Your changing the subject (again).

        Your post didn’t say “Nate Silver was Wrong About the British General Election” and I wasn’t defending Silver’s predictions of the British General Election. To use the BGE results now in defense of your critique of Silver’s analysis of the US Presidential Election is entirely specious.

        I expect Silver to get most of his raw state predictions right, 47, 48 or 49 out of 50, or maybe even a full score but there was a lot of chaos and complexity bubbling under the surface that so very easily could have changed the result.

        If you’re going to concede that he got 95%-100% of the State level elections correct, then you should write a blog post that is titled

        Why Nate Silver Was Right (Despite Chaos and Complexity),

        Otherwise, you’re just being stubborn.

        • The “wrongness” I am referring to is not paying sufficient heed to the chaos and complexity. The tails are fatter than his model puts them. On this occasion, the election outcome was Mediocristani and so his model looks vindicated as the effects that I identified were not large enough to push the result into unexpected territory. But — and this is crucial — it would not have taken a much bigger push to send the results wildly out of order, as happened to his model’s output during the British election in 2010 and his prediction of a sixty seat (+100%) gain for the Liberal Democrats turned into a 5 seat loss (-10%). As I say electoral politics has very fat tails, and Silver’s model has thin tails. Let him enjoy his “vindication”. Another day it will be different.

        • My model doesn’t just give 50:50 estimates. Earlier in the campaign (say before the first debate) I would have heavily favoured Obama by at least 65:35. I gave a 50:50 estimate because I saw polls estimating the national vote as close to tied, and because while I saw strong reasons motivating Obama supporters, I also saw plenty of Obama weaknesses in the possibility for a Bradley Effect and Shy Tory Effect (based on historical data), and the possibility that polls were actually overestimating Obama’s strength as an incumbent (also based on historical data). It was a finely balanced election, but in the last few days Obama grasped the momentum, pushed onto victory and justified his state poll numbers. Nate Silver’s model has fallen down on these problems in the past, too (e.g. Britain in 2010).

      • The results are a lot closer to the model that gave 91-9, given, you know, that Obama took all the swing states save North Carolina, which was pretty much what Silver was predicting.

        • Yes, but the 91:9 model does not in my opinion account adequately for tail risk. Human behaviour in polls is inherently unpredictable and unreliable. The state-level polls in this election performed excellently, but sooner or later the chaos will rear its head again, fragility hunters will be rewarded.

          On the other hand, if the choice is Nate Silver who chose 91:9, or Sam Wang’s 99:1 model, Nate Silver’s is far superior because it at least is open to some degree to the polls being wrong. The difference was, I thought, that in this specific case, Silver was still being overly-optimistic. It is uncertain who was right and who was wrong in this case (because we have a sample size of 1 election), but I will always err on the side of caution, i.e. other analysts are underestimating tail risk.

    • That’s what they said about Myron Scholes…

      Fortunately for Silver, elections don’t come around often enough to show just how fat the tails are….

  14. nate silver went 50/50 on the states…and his highest prediction in the model was 332 electorals for Obama…he got 332..on the nose!

    • Crying? Everything that happened was totally consistent with my preconceived model. I said above that this kind of result was likely, in fact. Doesn’t mean Nate Silver didn’t underestimate the tail risks (he does, that’s why he screwed up the 2010 UK election predictions).

      Also, I am immeasurably happy the Republican Party is imploding:

      Next time maybe they will bring Ron Paul supporters and other civil libertarians on board. In that case, they might win.

    • Personally, I think most of that board except the ridiculous projections from the likes of Cramer and UnskewedPolls was “possible”. My whole point is that there was far more uncertainty and chaos underlying the whole thing. I see the range of possible results as a far broader bell curve than most analysts, much fatter tails. Good day for Silver’s book sales, though.

  15. I take your point about the Bradley effect, having lived through it in Virginia (when Doug Wilder beat Marshall Coleman for governor in a race that was far closer than polling prediucted) but maybe I don’t understand your terms. I don’t understand what “a range of possible results” has to do with predicting the actual result, or what “fatter tails” means. I read what you said about the failure of his model during the British elections; could he not have learned from that embarrassment and refined his model?

    All in a day’s work. In any event, you often post provocative and thought provoking analysis. Keep up the good work.

    Surly1
    http://www.doomsteaddiner.org

    • Nothing that happened was at odds with anything I said.

      In reality, the tails were far fatter than Silver’s model. That the end result was toward the middle of the range of possible distributions says NOTHING about the tails.

        • People slagging me off here don’t understand my post, or Silver’s work. Silver doesn’t make predictions (nor do I) he gives probability distributions. He hit this probability distribution right at the bell. That says nothing about the tails. In the end, people will make a lot of money betting against Nate Silver’s thin-tailed distributions, especially now they will be taken as gospel on Intrade and Betfair.

  16. http://www.milupa.it/speck5.com.htm ma non stato detto fino a subito prima dell’imbarco. Ha cercato di usare il bagno prima del volo,ma il suo corpo non ha collaborato ..Top Regali ragazze dovrebbero avere il Top 8 regali per le ragazze. Fondamentalmente sono la migliore vendita di oggetti rosa da diverse marche di computer portatile, telefoni cellulari,Cintura Burberry, borse, e macchine fotografiche. Essi sono indicati come i migliori regali di donne rosa dovrebbero avere, che un sacco di donne hanno acquistato e fatto commenti da regalo migliore rosa per ogni voce di rosa .
    repliche borse Burberry Canvas Series http://www.milupa.it/speck5.com.htm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s