Measurement: time to look up, not down

Measurement: time to look up, not down


Better metrics and frameworks aren’t the primary solution to public relations measurement. We need to ask better questions first. By Nigel Sarbutts, @NigelSarbutts

In this guest post independent consultant Nigel Sarbutts talks to a client-side head of communications and a procurement expert about why public relations measurement is too focussed on dashboards. He sets out a vision of what needs to change if measurement is to demonstrate the strategic value of public relations.

As so often, it started with a tweet.

It’s time for public relations measurement to put down the spreadsheet and look up and accept that its focus on frameworks and dashboards is not working.

In making this claim and in setting out the way I think that measurement has to change direction to be more valuable and relevant, I have sought the client perspective from Jon Sellors [@jonsellors], head of communications at LV; and Steve Ashcroft [@thinkprocure], an expert in procurement.

I do this not only because they have valuable opinions, but because the solution to PR’s measurement problem has to be one where both buyers and sellers of public relations recognise that measurement is not a negotiating tool towards higher or lower fees, but that all parties have common ground in demonstrating the strategic value of public relations.

Thesis 1: measurement is not an end in itself


Public relations measurement is and always has been focused on data points at the end of a process. It has become more sophisticated and the data points have become more diverse and numerous, but it is ultimately what engineers call an “end of pipeline solution” – trying to fit better filters to the outflow, rather than looking up to work out why this stuff coming down the pipeline is the colour it is in the first place. The opportunity is to be better at looking upstream and ask ‘why’ questions, not ‘what’ questions.

Jon: I Totally agree that public relations measurement has been and is still focused on the end of the pipeline. Evaluation reports have certainly become lengthier covering more data points but I’d question whether they are more sophisticated.

My frustration with media evaluation suppliers has not changed over the years - they are very good at producing a pack of pretty graphs but is there anything actionable? I’m ashamed to admit that I can’t think of a time where I’ve changed strategy or plan based on intelligence I’ve received from my media evaluation. So why do I keep paying for what’s little more than an expensive comfort blanket? Probably because we’re told you must measure.

Thesis 2: Despite their flaws, AVEs persist for a reason and we are not asking why measure public relations not media relations


The situation, as evidenced by the latest spat over advertising value equivalent (AVE) (and the fact that after so many years they even merit a mention in the latest version of the Barcelona Principles) says to me we are long past the point where we should challenge the public relations measurement sector (the systems vendors and various measurement panels) to practice what they preach and measure their effectiveness in trying to get buyers of public relations services to adopt the Barcelona Principles – my hunch (backed up by judging a big public relations awards scheme for two years and the briefs I see) is that they simply don’t appear on the radar of clients and are not really observed in practice. 

Jon: Yes again. The AVE debate has raged for many years but the fact that it still exists shows something more fundamental is wrong.

We can’t keep saying they are not appropriate and expecting the situation to change. As well as challenging the industry to practice what it preaches (and I saw an AMEC member and potential new supplier recently and AVEs were still included in the sample charts they gave me) I wonder if we need to challenge what is meant by the ‘public relations measurement industry’, it needs to move on from just analysing coverage.

My take from judging awards is that measurement is included as an afterthought and goals or targets were not set at the outset hence so many vague comments such as “the client was very happy” and “lots of coverage was achieved.”

Thesis 3: measurement frameworks don’t allow for the agile and iterative nature of campaigns


Frameworks and principles which can be boiled down to “establish the business goals and then measure success towards them” are fine as words on paper, but ignore some very serious obstacles in practice (this is not an exhaustive list):

  • What you describe in a brief and how I interpret that in terms of measurable results can be very different – phrases like “think outside the box” are highly subjective
  • “No plan survives contact with the enemy” (ie. a sceptical media/audience that cannot be bought, or their response reliably predicted)
  • Shit happens and long term goals are prone to getting chucked out when the next quarter’s figures look unattainable
  • A lot of briefs are plainly cut and paste jobs where several different people with different desired outcomes/agendas are all convinced they have given a precise instruction set and will have very different ideas about what success looks like.

Jon: I would add another element in here. I think public relations agencies and measurement suppliers need to help in-house teams educate boards/exec teams about the role of communications and therefore what should be measured and reported back to them.

If a senior stakeholder is used to someone giving him/her a monthly AVE, reach or share of voice figure we need to effect change around the exec/board table first.

I don’t think that the public relations measurement industry can exist in isolation. People don’t consume media or form impressions of brands in isolation. Measurement has to be much more rounded and sophisticated. I’m increasingly looking at overall measurement of reputation rather than analysing coverage.

While there has been much talk over several years about the blurring of boundaries between public relations and marketing, driven particularly by the growth of digital and social channels, I think the new blurring is with public affairs and corporate responsibility. Hence why I’m now more interested in measuring reputation and doing this across various stakeholder groups.

So, what’s the solution?

We should be bold and completely change the public relations measurement debate from the end of pipeline pursuit of ‘better metrics‘ and theoretical frameworks, to the pursuit of ‘better questions to define the problem to be solved, at the outset’.

 That would have three strands:

  • buyers and vendors of public relations services collaborating on best practice for RFPs and ongoing campaign management in writing a brief in which objectives are described in precise, unambiguous language and written in a way that anticipates both variations and measurement.
  • collaboration on standard models of payments and incentives which force buyer and vendor to be precise about communications objectives and precise about how a midstream deviation from objectives is agreed and incorporated into the risk/reward in the contract
  • buyers and vendors collaborating with procurement specialists to agree how much weight is attached to cost in a contract and how much to measurable outcomes

Sounds nice, but it’s not easy so how could the industry develop a more collaborative, rather than “go on then, prove it” way of thinking about measurement.

Here’s where the opinion of procurement expert Steve Ashcroft [@thinkprocure], who blogs at ThinkProcure, comes in and his response to the challenge of how do we deliver against these three strands  is a set of questions for both buyer and vendor of PR services which would go a long way to establishing measurement and collaborative best practice – leading up to the big, abstract question, posed at the end.

Steve:  If I was advising a buyer keen to explore a more collaborative procurement of PR services, here’s where I’d start. A little bit of humility by asking questions and listening – carefully – would be a good starting point to establish co-operation and, heavens, maybe even trust.

Q1. Do you the client know what you are buying? Admit you are not (always) a technical expert – get the PR services provider to explain their value in business terms.

Q2. Are the responsibilities clear (who, what, why and when)? Let’s be clear on obligations and liabilities (note you, the Buyer might have some).

Q3. Has the PR services provider explained how the deliverables are going to be achieved? A method statement or plan (you did ask for one, right?).

Q4. What does the PR services provider need from you? Dependency on the Buyer can we a useful excuse for failure – by some PR services providers. Clarity is key.

Q5. Are deliverables linked to acceptance prior to payment? You need to define and jointly agree what is acceptance (agreed prior to contract award).

Q6. What happens if things go wrong? We always ask the (big-spend, strategic) PR services provider for their risk register (they have got one, right?).

Q7. Are responsibilities to report progress clearly stated (no surprises)? Ask the PR services provider to define Management Information in their Proposal. (I am aware of the Barcelona Principles, let’s see if your PR services provider can articulate how they will be reported (and delivered against) on your project.

Q8. How is the PR services provider’s performance monitored? Is everything on track to ensure delivery? How do you know?

Q9. What will success look like? (the quantitative values as well as the qualitative values (key messages; mindshare; influencers, customer or analyst credibility). Clicks, conversions and sales are value, of course! Can the public relations service provider articulate success metrics? Are these clear in the contract?

Q10. If it is not clear to you, how can it be clear to the public relations services provider? (Admittedly a question for you, the Buyer more than the public relations services provider).

What is ‘it’ in Q10? It is the complete procurement – the brief, the metrics, pricing, contracts obligations and liabilities – the whole relationship in fact.

Misaligned expectations are the number one cause of commercial disputes. Buyers – try out the questions – and for public relations service providers on the other side, are you sure you can evidence answers convincingly?

Moving on, and up

Goal congruence between buyer and vendor through a collaborative, trusting relationship with evidence-based performance visibility is not even remotely easy, so why bother?

The prize is enormous and a shared one for buyers, vendors and procurement people – clarity, standardised practice to enable valid comparisons and an absolute focus on addressing a business’ true objectives in a way that is recognisable to the board.

The measurement debate has been useful and valuable, but we need to move on and move up.

If you think that’s worth exploring then let me know and let’s get some smart people in a room.

About Nigel Sarbutts

nigel sarbuttsNigel Sarbutts has a 30 year background in public relations consultancy with managing director roles at Burson Marsteller and other leading organisations. He advises clients on communications strategy. Contact Nigel @NigelSarbutts.

30 alternative public relations metrics to AVE

30 alternative public relations metrics to AVE

Highly recommended

Highly recommended