This originally aired as episode 50 of The Assurance Show.
In this episode, we discuss the benefits of adopting a data focused approach during audit planning.
Conor:
Hello and welcome to the performance audit report. My name's Conor McGarrity.
Yusuf:
And I'm Yusuf Moolla. This episode is an older episode. It comes from The Assurance Show, which is a podcast that has been going for a little while. That podcast was focused a little bit more broadly. So internal auditors, performance auditors, and a few other people that have primarily data as a focus in their work. But this episode is specifically performance audit related. There's about 10 of them that will be putting onto the show from time to time. They are a bit older. Some of them are interviews with performance audit professionals in Supreme Audit Institutions. They will be relevant directly relevant to you as a performance audit report listener. Just to know that they are a bit older. So if you have listened to the assurance show before you may have come across this episode, and so apologies for the duplication, we just think that it's worthwhile bringing that across, into this podcast because of its relevance. Hope you enjoy it.
Conor:
We spoke about developing forward work plans, so covering your three-year horizon, let's say. Okay, so you've got everything settled, you actually need to start executing some of your audits. What are some of the things that you may need to consider in using data for the planning of those audits?
Yusuf:
Okay. So that's the next logical step. Now that you've actually selected your topics, you're going to go into a particular topic. Most people are very familiar with what traditional planning would look like, so research, and understanding the entity, understanding the topic. And what we want to talk about in particular is what, the use of data within planning can help with.
Conor:
Yeah. And so some of the benefits of a data focused approach probably haven't been articulated that clearly. Because generally teams just get into planning and they don't have time obviously to sit back and think of what else can be delivered using data in this process. We'll talk a little bit about that today.
Yusuf:
And the proliferation of data, particularly open data and obviously a range of different corporate datasets, means that you could take this approach for pretty much any performance audit. So what are the benefits of using data within planning?
Conor:
The first one is obtaining the data early. That allows us to explore the performance of the organization to try and understand what our key audit questions might be. Things that are possible to be answered in this audit, but also equally importantly, it allows us to zone in on things that we may have to exclude only because either the data doesn't allow us to answer those questions, or we might have to go to another source or it may be too time consuming or complex. So it allows us really to frame up our scope of what's possible, and explore how we might during the conduct phase of the audit go about answering those questions.
Yusuf:
Potentially using data to identify, like you said, a sample, but also scope exclusions. What would that look like? So how do we use data to identify a potential scope exclusion?
Conor:
So one example springs to mind straight away is where, we may be able to use the data that we have to half answer a question, but we may have to get extra data from another entity to fully answer the question. Now, it's very important to realize that upfront because there may be limitations to actually engaging that other entity be able to fully answer the questions. There may be constraints around time, to do that. And there may even be prohibition or legislative constraints around asking for that data. That's a really important constraint that having data up front allows us to understand so that we can potentially exclude that from our inquiry.
Yusuf:
Does that mean that we're excluding it on the basis that we can't get the data. Because, surely we can then get the information or ask the questions regardless of our ability to actually get the underlying data, or is it about how deep we can go on that particular matter?
Conor:
It's more the latter about how deep we can actually go. Because, to be quite frank here, it may well be that it's not worth the extra effort to be able to answer this question completely, in the grander scheme of the audit.
Yusuf:
And so what if the data tells us that there might need to be an exclusion, but excluding it means that we're not able to answer a fundamental question that is being asked in the audit.
Conor:
That circumstance would put in jeopardy the entire objective of our audit to be able to conclude reasonably on that objective. So either we need to go back and revisit the objective of the audit and recast what we can actually do. And that may well mean that we can't provide a conclusion to the extent that we wanted to. Or it may well mean that we have to completely change focus. If that missing piece or that exclusion that we've identified, if that is so substantial, that it means we can't meet the audit objective, then we'll have to entirely change that objective. But if it's more about how we go about getting evidence cause data really is just evidence. Then it may not be so significant that we can't obtain that evidence from another source.
Yusuf:
In summary, what we're saying is that using data, upfront during planning, helps us to properly answer the questions around what we can do, what we should do and how we should do that. So we're better informed during the planning stage about the direction and the approach that we are able to take based on the information that we already seeing, rather than coming up with what we think we're going to do, and then finding during conduct that we're not able to do that.
Conor:
What's possible. What's not possible. The extent to which we can answer questions. Providing us with a defensible basis so that when we're asked later on, why did you guys exclude this from the performance audit? We can say we looked at some data to help us form the scope of the audit. And these are the reasons we slightly changed tack, or we modified the objective.
Yusuf:
And we were able to go and look at alternate sources. We were able to look at alternative approaches and we aren't reasonably able to answer that question.
Conor:
Auditors General, who are attending parliament to talk about their performance audit quite reasonably, they may get asked the question: How come you said in your forward work plan, your objective was this and then it modified. So it gives the defensible basis for them to say we looked at the data and we had to change the objective slightly.
Yusuf:
Okay. So that's about the overall objective of the audit and, broadly some of the exclusions and the approach that we're going to take. What else does an early focus on data help us to do during planning?
Conor:
We've spoken at length, about the duration of performance audits. And some of the shorter ones are around the six month period, and they can go up to beyond a year. When you think about that in real terms, the world's moving so fast. Your audience may lose sight of what's actually going on from commencement to reporting. Now, one way you can maintain their engagement and enthusiasm and give them a product or something to see as an outcome of what you're actually doing until you table your report, is through these ancillary or byproducts or call them what you will, that you are creating during the audit based on your analysis of data.
Yusuf:
So that's providing a dashboard or some sort of other insight early on during the performance audit.
Conor:
Absolutely. So let's assume that a performance audit takes nine months, let's assume that your planning takes a couple of months and as part of that, you've grabbed a whole heap of data and you've done what we just previously discussed. You've done your exploration of the data and you've had to put it into some sort of shape and visualize it to see what the story is. Now, if you can, at the end of that planning phase, even before you move into conduct or even in the middle of conduct. If you can clean that up, that analysis and make it digestible, and in a format that members of the public can consume, or the parliament can consume in the intervening period, as you do your conduct work, then surely that's a benefit to all those stakeholders who are really interested in that performance audit.
Yusuf:
Okay. And what about the entities? Is there potential for putting information out that the entities can consume, understand and use to potentially fix up some of the issues that may be identified before the end of the audit.
Conor:
So that would be an ideal scenario. By the time an Auditor General came to table the report in parliament, all the issues identified during the course of the audit would have been fixed up quick smart by the entity. And we always talk about efficiencies. And that's an efficient system there in and of itself such that the prompt has been the audit in itself. If something is published or released partway through that. And it could be, like we said, based on that analysis of data that has illuminated the problems or the issues, they have then been fixed up in the intervening period. Then surely that performance audit, regardless of being tabled in parliament, has created value for all those stakeholders along that delivery chain.
Yusuf:
So an example of that might be where we take some data early on from the entity. And we understand what the data quality gaps might be. Now if it's purely just data, quality gaps in the work that we're doing. That's one thing because that's about ensuring that the data is clean, but where we're looking at things like how data is used for monitoring performance, managing performance. If the data itself is not very clean we can then highlight those data quality issues because that will directly relate to the ability to monitor performance. And so if we can show what those potential dirty data issues are and the entity is then able to fix that up. That means that before the end of the audit, they're actually able to improve the way in which they monitor performance based on that underlying data.
Conor:
That's right. And just one more point on data quality, is that a lot of entities with the best will in the world want to capture performance data. A lot of the times they rely on other government departments or agencies to provide that performance data to them so that they can coalesce it and then bring together overall performance. Their focus through that process may not have been on the quality of what they're receiving. So if an external organization, like the audit office can say, these are where your specific deficiencies in data quality lie, it gives that agency the ability to go out to those other collection points and say, these are the specific matters that are deficient in terms of your data quality. We can't accurately monitor performance of this program or of this activity until you help us fix those up.
Yusuf:
The first one, we said there was ability to frame up properly, our objectives, questions, lines of inquiry, scope exclusions. The second one was providing some data to ensure early insights in what normally would be a long process and potentially fixing some of those up. What else can we do with data early on to help the way in which we execute our audits, either internal benefits or benefits out to the public and parliament.
Conor:
The expectation among the general public and even parliaments now is that there will be data readily made available to them to consume. So we've seen the rise in data journalism, for example, during COVID where there's graphs and charts and statistics and all sorts of data visuals appearing in newspapers and TV screens. So the expectation among the public one would suggest is growing around data and those visualizations. So their appetite probably growing as well as their ability to actually understand what's been put in front of them. So what we need to do as auditors is make sure that we are feeding that appetite, and there is a large amount of information that we have available to us, through our performance audits that can help us in that respect.
Yusuf:
Third benefit is, our ability to continue to be relevant in relation to what our stakeholders are looking for, what their changing needs are, and how we can address those. So the last question I want to ask then is, you've got the traditional, performance audit objectives that focus on efficiency, effectiveness, compliance, and economy, as everybody knows. But what we're seeing more and more is fake news, incorrect information, poor data quality. Lots of information that is being put out, particularly, as you said now with the pandemic and the continued rise of data journalism, or maybe the way in which data journalism is now potentially far better understood by the public, who have an expectation to consume some of the data and understand what it means. Do you think that there may need to be change in exactly what an audit office does. Open data, publicly available data we've seen over the years, that It's not always consistent, it's not always very clean. Is there a role for audit offices to either separately or as part of the audits evaluate the quality of open data that is being provided and used.
Conor:
So the answer is yes. I think it goes broader than just evaluating the quality of Open Data. There's a growing pattern of Auditors General being given hard questions by parliaments and members of the public, call them referrals or requests for audits. They are getting more and more of that sort of work because they are viewed as an honest broker about what is real information and what is not real information. There will be a growth in that type of work coming to Auditors General, which will change the way in which they do their work and their focus on other things like compliance, economy, efficiency and effectiveness. And they'll have to allow for that honest broker role as part of their overall work.
Yusuf:
So if we said add a fifth objective. Maybe we'll call it T for transparency, which everybody is asking for. Which may fall into one of the other four objectives, but may actually be a separate objective in itself. What's the opportunity for something like that, without changing an audit office's mandate necessarily.
Conor:
There's a fair bit in that sentiment there. So the first thing is transparency should always factor into those four main objectives. We talked about exploring data to try and get transparency for ourselves about what we should look at, we've talked about producing data visuals, for example, through our audit that provides transparency as we go along. However, the second and bigger, and probably more strategic question around whether or not transparency should be part of our mandate as a fifth limb, I think that's a matter for debate and each jurisdiction will be different, but how does that impact the way in which Auditors General think about their work? There's probably scope there for, when they are formulating their audit work program and they're trying to pick which topics are more meritorious than others. It may be an interesting question to ask, when we're thinking about those topics, which of these topics do we think has the least amount of transparency attached to it? So if it's a government program, for example, maybe we add a metric or make some evaluation around Is this a program that we don't really know anything about or there's not much public information. Would conducting this performance audit really enhance transparency. So perhaps that's something that needs to factor into that planning discussion.
Yusuf:
Performance statements within the public sector are being evaluated by some Auditors General. They're looking at the way in which service, delivery statements or performance statements are crafted and whether the information is accurate. So is that then a separate matter or a separate type of topic that can then be applied to other similar domains.
Conor:
the transparency of information and integrity of information combined together in the same sort of stream. So the abundance of information, or like you said, the proliferation there, now, perhaps we have a stream of work within our forward work program that is entirely based on looking at transparency and integrity of information. So like you said, there's a whole heap of information being put out there, but we're through our performance audit work program going to validate whether or not it's transparent complete and accurate.
Yusuf:
A number of matters that we spoke about there, but to summarize the first thing is using data as part of our planning can help us properly inform our objectives defend what it is that we're going to be looking at, not looking at, understand our approach, understand scope exclusions, and go deeper than where we would be if we were only to start to grab data during conduct, if at all. The second was around providing transparency. Or providing early indications to the public, to parliament and also to the entities, to potentially fast track the remediation process for any findings that we may have later on. So the third one was audit offices meeting people where they are. Understanding what people are already looking for, and starting to meet that expectation of publicly available information that they can use and understand and consume in an easy way. So graphs and charts, et cetera. And then lastly, we spoke about transparency and integrity being something that we've been doing as audit offices all along, through financial statement audits and performance audits. Potentially there might be an opportunity to increase what we do in that area. To not necessarily just hold governments to account, which is something that we do obviously, but also to enhance the way in which information is produced, and enhance the way in which information that the public rely on can be trusted. So improving trust by the public, in the information that is produced by government, via an audit office stamp of approval, which is something that is always highly sought after.
Conor:
Cool, thanks Yusuf.
Yusuf:
Thanks Conor.
