It’s a common misconception that consultation is a vote but even those “in the know” realise that a popular outcome is the best way to justify a change. Focusing on the numbers is not entirely helpful; questions should instead be raised about representation and the quality of responses. When numbers come under scrutiny, independent analysis is a first line of defence – when it comes to upholding integrity, “marking your own homework” is never a good idea.
Moreover, the numbers aren’t real until a final analysis is done. Many a politician have been caught out declaring mass engagement only for the numbers to be corrected at a later date due to fakery. Given that “fake engagement” is the new normal, it’s quite reasonable to separate out duplicate or “campaign responses”. This can make the process of analysis much harder. True rigour involves making multiple analysis, such as with and without any offending items.
When an unpopular decision is made against the numbers then the tactic should be to explain why a decision was taken and look for compromises. A duty to explain for planning (on refusal) is normal but the narratives provided around consultation & engagement decision-making can be poor. In England, the “Openness in Local Government Bodies Regulations (2014)” stipulate that: The reasons may be briefly stated but must be proper, adequate and intelligible and deal with the main issues and substantial points raised (including objections). If decision-makers disagree with respondents about the impact of proposals, they must explain how they have arrived at a different judgement.
The real problem is disclosure. Very few statistical reports talk about the “degree of confidence” in consultation & engagement results or even publish details of numerical fixes. Even fewer consultations remind people that the sway of opinion is not binding. Unfortunately, many consultation reports have a statistical deficit. For example, quoting percentages without indicating total numbers in a sample or presenting statements with unintended bias.
And so, we come to an argument over the London ULEZ where Sadiq Khan has been accused of “falsely and dishonestly” manipulating the results. Part of the problem was that Mr Khan and his deputy Seb Dance, a former Member of the European Parliament, told the London Assembly that they had not been briefed in advance on the ULEZ consultation results when asked repeatedly by Tory assembly members at public meetings. Doubt over this was later cast in the publication of a report obtained via FOI.
Continual briefing politicians on numbers is actually a good strategy and mostly harmless – it helps avoid any legal claims under Gunning four (“The product of consultation is conscientiously taken into account by the decision maker(s)”). Quite why the Mayor and his deputy said they weren’t briefed is a mystery but perhaps they didn’t listen very carefully!
A claim of improperly excluding thousands of legitimate responses is serious but, in our view, unlikely. Some calibration did happen – but reasonably. For example, early 5,000 so-called “copy and paste” email responses opposed to the ULEZ extension – organised by the pro-motorist Fair Fuel campaign group – were excluded from a ULEZ YouGov poll. Besides, even discarded responses to the consultation were analysed and TfL have excellent pedigree in this area.
The sobering truth is that politicians must pay more attention to consultations as they unfold – and the accuracy of the data at the end. Consulting organisations must be more rigorous and open with how the analysis is conducted (perhaps even publishing the raw data for independent analysis). Meanwhile, stakeholders must let go of the numbers and focus in on any key arguments raised.
We’d advocate something even better to avoid this happening again – an industry standard format for consultation reports. It might even make mass participation data a reality which could help prevent fraud. And where do people go when they have a genuine complaint? As examples like this become more frequent, the case for a government “office for public engagement” is made.