Reclaiming the pride of research: Where Kenya’s pollsters went wrong
Political opinion polls are designed to be useful tools for predicting election outcomes. If carried out scientifically, an opinion poll covering a sample of as little as 2,000 people should correctly predict the outcome of an election within a very thin margin of error.
What this means is that if the first 2,000 voters on Election Day were a true representative sample of the voting population, how they vote would give a fairly accurate picture of the expected election outcome. All the other people on the queue would not really make a difference in the outcome of the election; they would just beef up the numbers. Obviously, it is never that simple in real life.
The Tyranny of Numbers
Kenyan pollsters have been accused of failing to give accurate results reflecting the political reality. In the aftermath of the 2013 general elections, their methodology was called into question.
If the polls had been scientific, different pollsters should have posted similar results within a reasonable margin of error for the same period. However, that was not the case raising questions regarding their objectivity.
In some cases, it was possible to predict the poll outcomes before they were announced; in other instances, it was obvious that there was some interference with the polling process.
With the election now behind us, evidently, the pollsters have been proven wrong While, prima facie, Mutahi Ngunyi’s controversial theory of the ‘tyranny of numbers’ appears to have been vindicated.
Whatever the explanations the pollsters give cannot cure the dent inflicted on their credibility.
In my assessment, Kenyan pollsters failed because of two main reasons. First, by not taking due cognisance of the realities of the Kenyan sociopolitical landscape and secondly, by employing a flawed methodology.
What's your take?
Do you think Mutahi Ngunyi's theory of the 'Tyranny of Numbers was right'?
Understanding the context
It is a well understood maxim of social science theory that knowledge must be contextualized. Therefore, in conducting any research exercise, the researcher must pay attention to the environment within which they conduct their research and especially the population to which they intend to generalise their findings.
It appears that the pollsters ignored the reality of the Kenyan sociopolitical scene in conducting their exercises. As a result, it is evident that they didn’t take cognisance of the triumph of identity over ideology in Kenyan politics. They therefore discounted a significant source of voter motivation.
It should have been clear to them, as it was to everyone else, that being a high-stakes election, voter turnout in core Jubilee and CORD strongholds would be high enough to impact on their predictions. They ought to have taken this into account during the polling process.
This is the one thing that Mutahi articulated well. Even though, I don’t agree with his arithmetic and I also don’t think that his predictions were right, I respect his confidence and ability to state the obvious though not to the pollsters. Even those Kenyans who haven’t been to school have an element of appreciation of the importance of ethnic identity to Kenyan politics.
An opinion poll should provide a snapshot of what the population thinks about a particular issue at a particular time. Given that we already know that these polls are about election issues and the ‘horse race’, let’s focus on the population.
The fundamental question regarding a population in a research exercise is the extent of its homogeneity. For a largely heterogeneous population like the Kenyan one, many considerations have to be made before attempting any research.
If we start with an assumption that Kenyan voters are rational beings, we can make the simple argument that their voting decisions are determined by certain things. In other word, there are certain things about people and their environment that make them cast their vote one way or the other. Secondly, it also implies that the people vote to maximise their benefit, whatever that benefit may be. This in turn implies that if we know these things in advance, then we can predict reasonably well how these people will behave.
In the Kenyan sociopolitical context, and in many other developing countries, identity is king. Anybody wishing to review questions related to political behaviour must surely pay attention to such an important fact.
However, a closer look at the voter behaviour amongst Kenyans as demonstrated by the results of the elections between 1992 and 2013 would reveal certain ‘irregularities’ in this thought process.
For instance, in 1992, the Luhya gave 41% of their vote to Moi, 36% to Matiba and 18% to Odinga. In 1997, they gave 45% of the vote to Moi and 48% to Wamalwa, their own candidate. In 2013, they split their vote largely between Mudavadi, their own candidate, and Raila. Why?
This implies that any attempt to study the political behaviour, while taking into account the ‘tribe’, must also understand the other considerations that voters make. Understanding these considerations will help us determine the attributes of the voter that make him swing one way or another and hence be able to create the various profiles of Kenyan voters.
To put it simply, without considering the ethnicity of the voter, any attempt to predict voter behaviour is flawed ab initio. However, an equally important consideration for any researcher is that ethnicity is not the only determinant of voter behaviour in Kenya.
In order for pollsters to be able to create a snapshot of the population and their attitudes, they need to select a manageable sample of the people that is designed to represent the entire population.
In order to select a meaningful sample, the researcher must have a refined understanding of the population. That is why the questions we asked above are critical for any pollster who is hoping to regain his credibility in the political market space.
An understanding of the nature of the population means that the researcher will be able to create clusters and quotas of people or segments of the population to be involved in the study.
For a research project of the nature of an opinion poll to be meaningful, very keen attention has to be paid to sampling. It is necessary that the sample be representative of the population for which we intend to generalise findings. This means that the sample should be a microcosm of the population. It follows then that if we are to have a good sample, we must be knowledgeable of our population and the sampling techniques we employ must be informed by this knowledge.
Who the pollster decides to interview will have a bearing on the outcome of the poll. The pollsters do not have an elaborate database of phone contacts. This exposes them to the risk of interviewing the same person several times. In addition, not all potential voters are reachable on phone. In the Kenyan context, a decision to use a mobile phone directory as the sampling frame is severely limiting.
The other methodological challenge that our pollsters need to address is the method of data collection. A fundamental reality of social science research is that there will always be a trade-off between quality, on the one hand, and time and cost, on the other. Every time a researcher chooses to save on time and cost, he will most likely compromise on quality.
The data collection method employed by most of our pollsters is Computer Aided Telephone Interviews (CATI). This method has significant advantages for research projects such as opinion polls because it saves on both time and money. It is particularly useful close to the election when the demand for polls is almost on a daily basis. However, the method imposes serious limitations to the quality of data collected.
To begin with, the sampling frame is by its very nature flawed. In most developed nations, they use the telephone directory but in Kenya it seems that the pollsters have created a database of some numbers which they use for these polls.
Also, it is well-known that people tend to lie in interviews for a variety of reasons. It gets even worse when they are interviewed on phone as opposed to face to face interviews as the interviewer cannot observe the subject.
Our various pollsters should consider the trade-off between time, cost and quality and either use face-to-face interviews or significantly improve their sampling frame. Their predictions may thus be much more believable.
Furthermore, any serious social researcher should be interested in more than just describing things. To justify the effort and the resources invested in a research process, the research should at the very least try to offer explanations for social phenomena.
Therefore, as they present the findings of their ‘research’, pollsters should also attempt to answer the question, why? They should attempt to search for the causes of behaviour and attitudes so that they can alert us, the consumers, of what to expect.
For instance, if indeed the Kikuyu people will only vote for a Kikuyu candidate, then we may want to understand what determines who they vote for when they have two candidates on the ballot. Along what basis did they split their vote between Matiba and Kibaki in 1992, Uhuru and Kibaki in 2002 but not Uhuru and Kenneth or Karua in 2013?
Without generating responses to such incidences, the research enterprise of the pollsters will at best remain a basic exercise of merely generating numbers that are not hinged on any serious understanding increasing the risk of failure.
In any case, research should be about generating knowledge so that we can identify possible causes of phenomena, predict their future trends and prescribe possible solutions. Research that doesn’t help us do any of these things isn’t justifiable on any scale.
Learn from success and failure alike
Ironically, the same pollsters got it right in predicting the outcome of the senatorial and gubernatorial races.
Our pollsters need to revise their methodology and treat opinion polls as important pieces of research if they want Kenyans to take them seriously. They could begin by seeking to understand how the senatorial and gubernatorial opinion polls turned out to be accurate in this first place. The learnings thus gleaned can be applied to future presidential election and other opinion surveys.
They may want to revise their sampling so as to include all counties and factor in the underlying demographic realities. Kenya is not a homogenous society.
Kenyan pollsters have an onerous duty to reclaim the pride of research and opinion polling in particular, from the crude and rudimentary theories of simply adding up ethnic numbers to predict the outcome of an election.