Mike Scott is Lead Auditor and Manager Assurance and Audit at the New Zealand Department of Internal Affairs.
He has over 30 years of audit experience, including 15 years in performance audit with the New Zealand Office of the Auditor-General and 17 years with the UK National Audit Office.
In this episode, we discuss:
Links:
Conor:
Welcome to The Performance Audit Report. Today we've got a special guest with an extensive career in performance auditing, including leading performance audit teams in both the UK and New Zealand, as well as training performance auditors from various countries. He's also previously chaired the Australian Council of Auditors General heads of performance audit forum. These days he's based in Wellington, New Zealand. Still focused on public sector performance evaluation. Mike Scott, welcome to The Performance Audit Report.
Mike:
Thanks very much Conor and Yusuf. Really pleased that you've invited me I'm very happy to be here.
Conor:
Great to have you on board. Okay Mike, maybe can we start with your background, performance audit, all things in your world, during your career.
Mike:
When people ask me this question, I tell 'em, I've been an auditor, man and boy, if you like. Sometimes I think I was born an auditor. I joined the National Audit Office in London back in September, 1987. Which seems like eons ago now and probably is. And strangely enough, I joined with a view of being a financial auditor, joined their professional training program. But pretty soon after joining realized that there was this amazing world of performance audit, which I hadn't really known about beforehand. Like many people I thought about audit purely in terms of, adding up columns of figures and checking that they're correct, and, ticking them off. But pretty quickly became clear to me that, there was a much more fascinating part of audit, which was basically to look into how well organizations did things in the public sector and offer them some fresh perspective on how they could do things better. I did the minimum possible to qualify as a professional auditor with the National Audit Office and then very quickly spent the rest of my career with them for 17 years doing as much performance audit as I could. I studied chemistry at university. And actually did three years of chemical research before turning my mind to audit And I think that's of one of the great things about performance audit, is that it attracts people from a huge variety of different backgrounds. You don't have to be a mathematician or a statistician or an economist to be a performance auditor. You can be, but I've worked with performance auditors who are nurses, historians, archeologists, a whole range of backgrounds. Performance auditors can come from any sphere or discipline in my view. It's the curiosity that they bring with them, which is really what draws them to performance audit I think. So 17 years with the National Audit Office in the UK. Moved to New Zealand end of 2004. Really for family reasons. I've got two sons who are now grown up and, they were kind of, of an age there where we thought we'd like to share them a bit of the world. so we came to New Zealand. I was lucky enough to secure a job here with the, Office of the Auditor-General, and, spent 15 years. About 12 of those as the Assistant Auditor-General responsible for performance audit. Before moving to the other side of the fence, and spent a couple of years working with the police here, as the manager of their performance function, doing some fascinating work, actually with the police, kind of helping them to think about their performance in a different way. Police, traditionally have thought about their performance in terms of counting things, counting people, counting offenders, counting crimes. So we helped the police here design a performance framework, which was much more based on what difference they were making for the public in New Zealand in terms of improving the public's safety. Much more focused on what they deliver by way of outcomes for the public. And less in terms of law enforcement. And then, yeah, last couple of years, well, last six months actually have moved to the Department of Internal Affairs here, which is another fascinating organization to work for. Whole range of activities, from regulating charities, working with local government on the provision of water services in New Zealand through to, obscure things like management of ceremonial flags. And I think they might regulate dog licensing too, but fascinating organization in terms of variety of work. and I'm really enjoying the time here too. I've kept my hand in, in the last few years on the performance audit side, cuz um, I'm so passionate about it, to be honest. So I've worked with the Australian Council of Auditors-General, managing their training program for the last couple of years and also helping link together audit offices in this part of the world in terms of sharing practice and capability development.
Conor:
Thanks Mike. So you mentioned there, moving from the Auditor-General's office to the line agencies to help them, work on their performance, evaluation and improvement. Have there been any differences you've seen in terms of the approach you've used or skillsets you've needed between those two different types of organizations?
Mike:
Some, but actually a lot of commonality more than anything else. The discipline of, assessing the performance of an organization is pretty similar, whether you're looking at it from inside the organization or outside the organization, I think. You're looking for the same kind of attributes in terms of the way in which performance is measured. So, you know, the indicators of performance, more focusing on the output, and the outcomes end of things, in terms of what difference the organization is making, as well as, whether the organization is actually performing efficiently in terms of the way it does things. My reflection on moving from looking from the outside in to looking in out, would be that, it's reinforced if anything, my view that as performance auditors, it's really important to have that sense of realism in what we're actually recommending to public organizations in terms of what they can do in terms of how they can improve their performance. Recommending that the agency, puts in place a performance framework, for example, sounds like a pretty simple thing to do, but to actually put that into action, requires a significant level of investment and thought about what the outcomes are, which the agency is aiming for in the first place, which might not be that well defined. And actually need some considerable effort and, engagement within the agency to work out very clearly what the outcomes are. Once you've got your outcomes clear, thinking about, well, what are the levers, which the agency can pull internally to actually drive those outcomes. and how can you actually design some measures which enable the agency to understand how well it is using those levers and how it can do things differently to improve. So I think that the kind of realism and realism in terms of, offering practical, really useful recommendations is my overall reflection. I think, from moving over to the other side.
Yusuf:
You've seen performance audit from various perspectives two audit offices, for a long time. And then more recently, as you said, from the outside, but also a little bit from the inside, with ACAG. What would you say you've seen as changes over that period within performance audit offices and delivery of performance audits?
Mike:
I might start, Yusuf just by saying what I think hasn't changed. cause I think what hasn't changed in the time that I've been doing performance audit is the desire of audit offices for their performance audit functions to make a difference. And actually to add some real value by recommending, improvements to how, the public services can be delivered or by providing assurance to, parliament and the public that actually things are in good shape, and services are actually being, delivered pretty well. so it's that kind of making a difference. Hasn't changed and I think is really important, context. I think what has changed is In my view, the focus of performance audit has changed a bit. It's changed from where it used to be more focused. When I first started in what I call a kind of backroom, used to be kind of focused on is the engine of an organization running well, does it have the right procedures in place, systems in place. Are those systems being followed. Are they efficient? the focus now is, is the engine actually delivering. what the public needs, and is actually delivering, not just outputs, but it's those outcomes, is it actually making a difference to people's lives? That I think has been probably one of the biggest shifts. I've seen, in terms of thinking about and scoping. And focusing performance audits. And I think that's a really positive change cause, it means that audit offices are targeting their performance audit functions in a way that they have real, benefit, and value in terms of, change for the public, which is what performance audit is ultimately about. it's also brought about some changes in the way, the work is undertaken and the evidence that's gathered, it's broadened the range of evidence that I think, audit offices need to come to an opinion and view. If you're just looking at the engine, you can look within the organization for the information, the performance data, the evidence you need to actually assess whether the engine is running well. But if you are looking actually. Is the organization delivering for the public, then you're looking more outward to, how do we bring in the lived experience of the people who are receiving these services? And how do we capture that and represent that, in a way that it actually adds to the, value of the audit, because it can be very compelling and it tells a much richer story. If you can tell it from the point of view of the person on the receiving end, And it really adds a huge amount of insight,. And, cuz we can add that, service recipient view to what we know from what we see inside the organization. And we can put the whole picture in front of parliament in terms of how well this organization is functioning and how well it's delivering. So I think that's been a bit of a shift. The availability of data, is another thing and the power of using data, I think, lots of the early audits I Used to work on were primarily based on looking at documents, interviewing a few people, but there wasn't, anywhere near the quality, of data and information, that there is available, now, actually looking to that data and thinking about what that data can bring by way of added insight to audits is another area, which has changed. I know that now, an area of focus for most audit offices in this part of the world anyway, is I think in the recommendations in particular is largely determined by how well the auditor understands, the issue, and understands what the root causes are of the performance that they're seeing. there's a lot more focus these days and really digging more and asking those, why questions. I think there's a rule somewhere, keep asking why until you get the answer that you need or the understanding that you need. So I think there's just a bit more of an appetite, a bit more of a focus on actually following the trail right to, okay now we understand what is driving this performance that we're seeing. The other thing I think is just project management, on a more mundane, important point, these are major complex projects, performance audits. and they do require, project management, to enable 'em to go smoothly, both from a point of view of the audit office, but also from the point of view of the entity, but I think it's now been recognized more explicitly in terms of something which auditors need to be more skilled at alongside all those other good attributes that auditors have cuz project management is related to timeliness, and timeliness is related to relevance and the value of a performance audit is driven by its relevance too. if you've got an audit that's running on, then you know, you're losing value. The more it runs on.
Yusuf:
In terms of new evidence and availability of data, I might come back to that a little bit later, but you spoke about a shift from a backroom engine to delivering what the public needs. So shift from, internally focused to externally focused with a need for internal, stability and structure. For established audit offices, that might have been a natural evolution. When you work with, newer audit offices or audit offices that are changing the way they operate, what does that look like? Is there still a sort of, evolution that they need to go through in order to stabilize or, can they jump to the end?
Mike:
I think there is a bit of an evolution there. I do a bit of work with some of the developing audit offices. if I can call them that, who are still building their capability. And who sometimes have quite limited capacity too, in terms of being able to undertake performance audit. In those offices, if you are not a full-time performance auditor, I think it's more challenging, to actually take on some of the more complex service recipient outcome focused audits. and the temptation is I think, to focus on some of those more mechanistic audits, which have value, and I don't diminish that. Those audit offices, I think generally tend to start at those mechanistic audits. But we encourage them to add that service recipient outcome element to them where they can, because, it sounds challenging and it is challenging at times, but you can also do some pretty simple things, to actually capture that, outcome focus, view. Go and talk to, a few of, the people who were actually on the receiving end of the service, go and talk to some advocacy groups, who actually, are in contact with people receiving service and kinda get their view. It will enable you to come to some more specific and insightful recommendations. It's important that, you do those more regular audits well, but I think it's also important to be ambitious and kind of trying to give them some extra insight where you can, and it's easy to do.
Yusuf:
So talking about, discussions with, other stakeholders, there have been a few in recent years, strategic audit plans, forward work plans. that, have come out from audit offices where they'll involve a survey of the public about what's important to them, or have the strategic plan out for comment. So a draft published and then out for comment to the public and then sort of back in. Do you have any experience with that and what are your thoughts, on that.
Mike:
I'm a big advocate of it, Yusuf. I think it's really important to take that temperature, about what matters to the public, and for audit offices to be informed by that when they're thinking about, where to place their performance audit effort. Auditor generals, have the independence to decide, how they deploy their performance audit effort. But the better informed that is. The more likely it is to focus on the really high value areas that are gonna be of most interest to parliament and the public. There's a bit of tension there for audit offices because You don't wanna raise the expectations of, the public, too much by, saying that you can look into everything that matters to them. You need to get the engagement positioned right so that you're kind of managing the expectation, but at the same time, as involving the public and I believe you can do that. The audit office here in New Zealand has just put out it's work plan. and I believe that for the first time they did a representative survey, of the public. I think they surveyed about a thousand members of the public, about what mattered to them. and they've put in an audit, I believe in their work program, because it was really what came through from that consultation with the public about, an issue that they were really concerned about and want to have some assurance that it was actually being managed and tackled by the public sector in a good way. You just need to be careful in managing the expectation and the messaging I think, but, I'm all for it and think it really helps to target work well.
Conor:
Mike, I wanna, shoot forward in the audit chain and talk about when we get to a reporting stage in our audits. Now, one of the things there's been an extreme focus on over the past few years, probably, encouraged more by COVID and so forth is maximizing the impact of our performance audits to make sure that we've done all this great work, we understood what's important to the public but we still need to sell the message or we still need to maximize that impact. What changes have you seen over your time in terms of strategies to maximize that impact?
Mike:
There is a lot of emphasis on the report and reporting, which is right. That is the main output of a performance audit, but actually, I think, the impact is built during the course of doing the audit. I'm a great believer that if you, take people on the audit journey with you, then you will get a really high impact audit in the end of the day. And by that, I mean that, engaging, and talking with the agency that you are auditing right from the start in terms of involving them in your thinking about how you're gonna scope the audit. Right through to your evidence gathering and reporting, and along the way as, you are kind of building your view of what you're auditing, there are lots of opportunities to share with the agency and the management of the agency, what you are thinking. You know, in a way that you are not committing to a final view, but you're just reflecting back to them. This is currently where my thinking's at based on what I've heard I've seen this happening or I've heard this from people, you know, were you aware of that? What do you think? So just having those really important conversations during the course of the audit is where there's a tremendous amount of impact, I think, which is sometimes overlooked. Because it's not just the audit report, which has the impact. It's the trust. It's the credibility, it's those conversations along the way, which actually means that the audit report is going to land well with the agency, and is gonna be taken seriously by them. And they're gonna actually commit to and implement the recommendations cause they felt involved. They felt listened to, and they can see that the audit report reflects the conversations and the understanding that has been gathered along the way. If you get to an audit report in the end of the day, where basically what you're saying in the audit report is that the agency agrees with everything and is actually onto it and they start to make improvements I think that is a sign of huge impact. You don't have to have an audit report, which says, everything's a disaster and this is 12 recommendations, which need to be implemented in the next six months otherwise things are not gonna improve. I think it's much more impactful to give that positive, constructive kind of story I think of real change happening as well as impact. But to answer your question more directly. I think, as well as the written report, and making sure that is clear, and succinct, and these days uses really compelling, headings, graphics to tell the story. I think putting a human face to the audit report is really important too, in terms of the messaging. I know some audit offices these days are producing videos, of, the auditor telling the story of the audit that's very impactful, because having the face of the auditor, and the person there telling the story of the audit, you pick up so much more, by way of, tone language emphasis, that adds to actually adds to the richness of the view so I think that's a really positive step. if I could say a bit about what I think might add to that. adding the view of the agency, that's been audited having someone from the agency talk about the audit report, alongside the auditor could be really powerful in terms of, just reinforcing the messages from the audit and the value the audit's had. We do that sometimes on the training courses I run, we run little sessions where we get, someone from an agency to come in and talk to, new performance auditors about what's important to them as recipients of performance audits. and invariably, they talk about openness trust, that relationship building along the way. and we've also bring in the auditor to talk about their perspective and just watching the dynamic between the auditor and the, people from the agency really does give you a, a sense of how important that relationship is to the value of an audit.
Conor:
One of the common criticisms we hear about audit generally, but also performance auditors is that, you came in, you did this thing to us. You didn't Understand our business you've called out what we consider, as a service delivery agency, reasonably minor issues that are gonna blow up and cause lots of problems for us. Do you see there's any need or any scope for performance auditors to better understand or have a better appetite for minor under performance or recognizing that versus major under performance?
Mike:
I think what you're touching on there, Conor is balance really. and I think it really is important for, audit offices to be balanced, in the way they report their findings. I think it's really important for them to, not just report, where there might be minor or major, underperformance but I've never done an audit yet where there's not something positive to say about something that actually, being done well. And I think it's really important for audit offices to balance their reports in that way. My experience is that that most audit officers now are just focused on what are the most significant issues in calling out under performance. And they get it pretty right. I think in terms of the way in which they highlight, those really high priority, high urgency issues, you'll not always get a complete agreement between the auditor and the agency about what is, and isn't, really significant under performance or minor under performance. that goes back to the importance of having those good conversations along the way, because it will flush out where the audit office and the agency is a bit apart, and it will provide the opportunity for there to be a good level of debate and conversation to give, understanding about why there might be difference of opinion and for the agencies and the audit office to come closer together, it's really important to spend that time before you actually get to a reporting where you're sharing in that important natural justice phase of an audit this is what our view is. And, and this is where our key findings are at this stage. What do you, as an agency think of this, what do you an agency think are some of the issues or context that we need to take into account that we might not have reflected so far? But ultimately, that's the judgment of the auditor or not about whether it makes it into the report or not. But I think having the conversation itself builds that trust between the agency and the, auditor and means that even if the agency doesn't completely agree with it, they will still have felt that they've had the opportunity to debate it. and they will take on board what the, auditor-general is recommending And on the minor discrepancies, that's where I think the value along the way, really counts for me, cuz I see those conversations during the course of the audit, as opportunities to just say, look, we found this, you know, we think it's really important that you think about this and, and it's not ideal. we don't think it's significant enough that we're gonna put it in the report, but nonetheless, we're giving it to you now and reporting it to you now so that you can actually do something about it
Conor:
so those critical discussions you have during the audit, so that you end up with a report that is balanced, and you've worked through the issues and you've gone on the journey with the auditee. Those negotiation skills, or, being able to put your point forward and listen to the other side Is that something that can be taught?
Mike:
I think it, comes with experience. and I think it, it comes back to having an open mind. I mean, I think it's really important for auditors to be open minded. particularly for, auditors coming fresh to performance audit and I suspect financial audit too There's a fear of getting it wrong. And a fear of stating your opinion, too early, if you like. That's a bit of a challenge that people just need to get over, because, yes, auditors need to be credible What auditors report, needs to be absolutely right. but. Along the way when you're having those negotiations or those critical conversations, you don't have to go into them, with a fixed position. I think you need to go into them with confidence that you've got evidence to support what your position is, but at the same time, you need to be open to listening to the view of the chief executive or the senior management team, about, how they think your view just might need to take into account some things that you might not have considered. Doesn't mean you've got it wrong. In fact, it means quite the opposite. In my view, it means that you are doing your job and responsibly, taking into account all relevant views, ensuring that you've understood what you've audited before coming to a final opinion. it's a combination of, an attribute that you're looking for when you recruit performance auditors. People who are open-minded, but a skill also that you build through experience. The more those conversations you have, the more you realize, the importance and the value of listening and compromise where compromise is needed
Yusuf:
Switching a bit from, Some of the softer skills, if you like to, slightly harder skills, you mentioned earlier that there's a broadening in the types of evidence, availability of data, the power of using data. there's lots of different approaches to using data within performance audit, particularly structurally within an audit office. You either have a sort of centralized data team that does all the work or performance auditors are skilled up to use data themselves. Which of those do you see working, or do you think there's, potential for both of those to exist?
Mike:
I think there's potential, for both to exist Yusuf and I've seen both working, you can be both a skilled performance auditor and a data scientist. and in fact, some of the skills are very complementary. Equally you can be a performance auditor who, is not skilled in data analysis, but, can work with a data scientist to actually explore the potential of, data. and it is, another hard piece of audit evidence if you like using data. but actually it does come back to the soft skills for me in terms of exploring its value, because, the key for me is auditors and data scientists coming together and talking. about the potential of data, and talking very early, in the audit, and to talk about, what the intended outcome of the audit is, what the audit wants to add by way of value and talking to the data science people about that. so that the data science people can understand and think about, well, how can we use and analyze this data and present this data in a way that's gonna add value to the outcome that the audit's intended to have. and I think a performance auditor coming from a non-data perspective, just asking those questions about, look, I'd really like to be able to do this. I'd really like to analyze this data in this way. Can we do it? And the data scientist is coming from the point of view of, well, okay. You know, you could do this and that, you know, meeting in the middle, you get a really good result. I think sometimes people think, they talk different languages, but I don't think they do. They just need to talk, about the objective and keep the conversation at that level rather than digging into the technicalities of it. It's kind of idea sharing, is what it comes down to me, both the performance auditor and the data, scientist sharing their ideas, and you get some really innovative ways of examining and using data.
Yusuf:
a follow up from that, related to people and skills. there's a, what is well recognized as a resourcing challenge globally, at the moment over the last few months, anyway, maybe even a bit longer, what have you seen in terms of, new ways or innovative ways of addressing that resourcing challenge? Recruiting, training and retaining performance audit stuff.
Mike:
I always think it's useful not to mention the word audit in your job advertisement. strangely enough. particularly when you're looking for performance auditors. Because there's something about the word audit, which, I think narrows your market immediately. I'm being a bit light about that, but actually there's a serious point in there. I've in the past advertised performance audit positions from the point of view of the opportunities they provide for people to really have highly variable career, looking at different issues. and an opportunity to actually, add value, to public services across the whole range of the public sector. we sometimes advertised it as the best job. you've, ever had that you didn't know about until now. to draw people in. Cos once you've drawn people in, you can sell the value, of performance audit as a career. You also draw in a much broader range of, candidates, which goes back to the very start when I was saying, that performance auditors can come from any background. What's key, in performance auditor is someone who's really curious, really wants to, understand, uh, new things. The one who's prepared to ask those questions, which other people might not ask, cuz they think, oh, they're really silly. But, you know, Hey, they're the best questions cuz they bring, out the best answers and someone might have never asked them before. And they really get people thinking differently. So someone's really curious, asks great questions. Someone who really is able to. build those really strong relationships with people, be really trusting and open, is able to draw that line, in terms of objectivity and independence as well. But really strong people people, and also people who can really, write their story, in a really compelling, way. They're the kind of general skills you're looking for. I think in auditors. And so you need to draw people in, who might not think of audit as a career, but actually once they find out about performance audit think, wow, I want some of that because it sounds really interesting. And actually, it's gonna give me a real opportunity to actually develop a broad range of skills and to add a lot of value. The other, reflection I would have I think it's really important to support, people in making good judgements, but also Making imperfect judgements, and mistakes along the way, cuz that's how people learn. it's not a disaster. it's part of the journey of doing a performance audit is to gradually build your understanding, and work out where your judgment is, and not give it too early. So it's really important, I think in terms of retention, just to provide those opportunities for people to use their initiative, exercise their judgment and, help them in doing that. Accepting that they're not always gonna get it right first time and that you can support them in actually coming to a final view in the end of the day.
Conor:
Some of those recruitment challenges may be being exacerbated because more and more audit offices are getting a performance audit or an evaluation mandate. So that remains a challenge. However, on the good side, that means there are more performance audit reports being published. Be interested to hear Mike, are there any reports recently or in the past little while that you've seen that are interesting or where topical issues have been addressed in those reports.
Mike:
The whole COVID, pandemic, has produced a whole swarm of audit reports if you like across audit offices, which, have been a bit different. It's meant that audit offices have become more agile, I think in looking at issues, some of the almost real time auditing there has been of, roll out of vaccination programs, and the effectiveness of some of the support payment arrangements that have been put in place. They're great examples of, audit offices, exercising a bit of agility and acting quickly and early to add a bit of value to help things go, right. rather than after the event point out where things have gone wrong. there's a lot of learning, for audit offices in those COVID. Response audits. And it's one of the things in, the Australasian Council of Auditors General, community sharing a lot of practice on recently One which is, not so recent, but I think I'll just kind of touch on because. I thought had a high level of value when I was in the Audit Office of New Zealand is a series of reports we did on the police about cultural change in the police, which is kind of a really interesting area to audit too. You know, it's gets in more into that outcome end of things, but it's also a bit of an intangible. How do you assess culture and an organization and work out whether it's changed? cuz the police had a Royal commission of inquiry on police conduct. a few years ago here in New Zealand, there was a whole raft of recommendations and the audit office was, asked to monitor the police response. so we did about four reports over 10 years, progressively looking at how is the police approaching changing culture and whether their culture has actually changed. I don't think there's many things you can't audit from a performance audit, point of view. There's some audits where you can more easily identify what your performance expectations are. and some audits where it's more challenging, it was certainly more challenging in the police culture space to work out well. Okay. What are the hallmarks of police culture that we're gonna look for but you can still as professional performance auditors. come to a professional judgment about what a reasonable set of performance expectations is in most situations. in the last of the four reports we did, the ultimate test for us was, there was a cohort of, people who worked for the police 15 years ago, who are still in the police. Let's talk to them. What was it like 15 years ago? What's it like now? And what are you seeing different? And that goes back to really talking to people who are the ones who are living the experience of cultural change or a service, by those conversations, we were able to get a really good sense of the way in which the culture has shifted, as well as talking to people outside of the police in terms of, what have you noticed has changed in terms of your interaction with the police? Interesting audit. and illustrates, that, even the most challenging areas are ones where you can do, a well judged, reasonable audit
Yusuf:
if anybody wants to reach out to you, or understand more about your work, or the work of ACAG, what's the best place for them to contact you?
Mike:
Yeah, look, you can find me on LinkedIn. Reach out to me via LinkedIn and I'll be very happy to follow up and help in any way I can.
Yusuf:
Excellent. Thank you very much for joining us on The Performance Audit Report.
Mike:
No, it's been a pleasure. Thanks very much. Yeah.
Conor:
Thanks, Mike.
