This episode featured guests from the North Dakota State Audit Office:
Allison Bader, Performance Audit Manager and Claudia Kuball, Performance Auditor.
In this episode we discuss:
Links:
Conor:
Welcome to The Performance Audit Report. My name's Conor McGarrity
Yusuf:
And I'm Yusuf.
Conor:
Today we're joined by auditors from North Dakota State Audit Office. Allison Bader and Claudia Kuball. Welcome to the show.
Claudia:
Thank you. Thank you.
Allison:
Thanks for having us.
Conor:
Pleasure. Can you kick us off and give us a little bit of information about your state audit office, the types of work you do, and any difference between what your audit office does and other state auditors.
Allison:
In the United States, the states have individual legislative bodies that meet once a year or every other year, and they write the laws for our state. Our State Auditor is authorized by state law to do audits of our state agencies, political subdivisions, and our colleges and universities. So in North Dakota, our state law allows the State Auditor to develop the scope of our performance audits. Other states may have a more formal process where their legislature can also determine what performance audits that they would like done and direct those to their State Auditor.
Conor:
Fantastic. Thank you. And can you just give us a bit of a background about your role and what you currently do at the audit office?
Allison:
I am the audit manager in the state agency audit division. So we audit the state agencies, do performance audits of them. I direct a team who does those audits.
Claudia:
And I am working as a medical auditor within the auditor's office, which is kind of a new role for the state auditor's office. I worked as a nurse for 20 years prior to joining the audit agency. So when I joined the audit agency, I knew nothing really about the audit process itself, so I had no idea what internal controls were. I, I, uh, it was a steep learning curve, but it's been fun and it's been a really good, interesting journey for me.
Conor:
So can we just dig into that a little bit. What was the driving factor or what was the prompt to bring in somebody with medical skills to the audit office.
Claudia:
Maybe Allison is the better one to answer that. But when I did start, I was doing Medicaid claims testing for our office. That was something that was done by the Department of Human Services at that time, which essentially were auditing their own Medicaid claims, and they wanted to bring that into the auditor's office. And so, the Department of Human Services trained me as to how we audited those claims. I learned a lot about the audit process. As I was learning, Allison thought there may be potential for different roles for me within the office, looking at performance audits. And so that's where that position grew for me.
Allison:
The auditor's office has been looking at bringing in different backgrounds and different degrees into our office to be able to audit areas where accounting degrees, which is the majority of our auditors, where they don't have quite the experience in some of those healthcare related areas and other programs that the state oversees and provides regulation over. And so the medical area was the first area that we tried to implement a different degree and bring that experience into our office, just because the state has so many healthcare related and human service related programs that they regulate.
Conor:
Fantastic. And are there other particular skills types that you think would be really useful for a performance auditor?
Allison:
Of course, IT areas, any information technology, data analytics. There's just so much growth happening in, data and analyzing data mining, that type of stuff. And, it would be great to grow that area more within our office.
Conor:
We hear that it's really difficult for audit offices all around the world to attract and retain staff. Is there anything in particular that you guys are doing in North Dakota to try and get these skills on board?
Allison:
We have seen a decline in the number of graduates coming out, with accounting degrees. And there isn't really a degree for auditing. So that's a lot of the hands-on learning in the work environment. In recent years, we have been going out and getting more involved in the education that's happening in the colleges, having presence there going into the classroom, just showing the different options related to auditing while those students are in the class and hiring interns, to come into our office and learn about auditing. And they are right there doing hands-on auditing work when they come into our office. And then also, yeah, just continuing to think about other degrees that we could bring in, looking for those critical thinkers, those problem solvers where they can bring the knowledge of their degree and prior work experience into the auditing field.
Conor:
We've had a few guests on the show from various audit offices and former senior auditors who say that audit's sort of got a bad name or it's got this label of, oh gosh, here comes the auditors It's, probably, a profession that's maybe struggles a little bit with that.
Claudia:
That's a general concept of auditors. Our communications department is awesome. They really are outside the box thinkers, and so they try to, make fun of and poke fun at some of these things. And so they had our State Auditor dressed as a dinosaur, as a a T-Rex, and, it was a social media post that said, auditors aren't scary, or, or something like that. So it was, it was lighthearted and fun.
Conor:
All these professions like yours is nursing can really lend a lot and bring a lot to the performance audit profession. So it's fantastic to hear that that's happening.
Yusuf:
What are the characteristics of performance audits that lead to high impact amongst your stakeholders?
Allison:
Those audit topics that get that impact it's the ones that get that attention from the citizens, the ones that are impacting a public service usually, public safety, that type of stuff, those seem to get a lot of the public interest. And then when there's issues in those areas, it just gets more attention. The news media picks it up. Recent audits that have come out that we've done that have gotten that type of attention. We did a Child Protective Services review and we found that the Child Protective Services wasn't making timely contact with potential victims of child abuse, and that face-to-face contact. That was a recent one. Also, we did one with our Department of Transportation, protecting sensitive information that's collected for the Real ID. And they collect sensitive information like birth records.
Claudia:
billing information.
Allison:
Billing information. Um, so passport data. So yeah, so protecting that information because they do retain some of it. That was another recent audit that we did.
Claudia:
A lot of our audit ideas, come right from news headlines. Our auditor is very engaged with the public as well. On our website, there's a n opportunity for public citizens to address topics that they would like to have seen as performance audits. We get a lot of engagement from the public and, just looking creatively outside the box too.
Yusuf:
In terms of, getting citizens involved. So you said your audit topics and maybe, individual details within your audit are responsive to what you're seeing in the media, or being talked about by citizens. What are the mechanisms that you use to encourage citizens to participate.
Allison:
There's a form out on our website to be able to submit comments, questions, that type of thing, audit ideas. And then we also have LinkedIn, Facebook, generally they can reach out to our communications or directly to our State Auditor and submit questions and concerns.
Claudia:
We have an incredible communications team, so they are instrumental. our audit reports, go live on our website. So the citizens of North Dakota, as soon as an audit report is released, they're able to look at those reports. There's transparency. We want citizens of North Dakota engaged and seeing where their dollars are being spent. Our communications team, release press releases. And so these really impactful audits, will get a press release. The press contacts, our office for quotes and comments. we've had press conferences. our state auditor has held press conferences to discuss some of these, audits. And then our social media presence is very valuable too.
Allison:
Yep. And we do the media releases before the audits are released. That also gives the state auditor an opportunity to put a quote in there giving attention to the importance of the audit.
Conor:
Among the fantastic reports that your office does, we had a look at one of the fairly recent ones into the Department of Health and the storage and handling of COVID vaccines. Can you tell us a little bit about that report
Claudia:
Going back to public input. Our office did have a lot of, inquiries about vaccines. We didn't really know where to start with this audit. Honestly, it was the vaccines were brand new to everyone, so they were brand new to the Department of Health. They were brand new to our office. Really they were brand new to everyone. and so we were kind of deciding what angle to go through with this and, everywhere you went in town, you know, main arteries of traffic or road, you would see a pop-up vaccine clinic. So, I mean, if you went to Walmart, there would be a pop-up clinic there. you could go to a drive-through site and get a vaccine. And so as a nurse in my background, I really thought about the storage and handling of those vaccines, North Dakota, we get, very varying temperatures. So our summers would be really hot. Our winters are very cold and, just knowing what I know about vaccines and how fragile they can be, and knowing how they need to be stored and monitored. That got me thinking about, you know, well, how are they keeping these vaccines at the proper temperatures at these pop-up sites? and so that was kind of the angle that I proposed to Allison and, we decided to run.
Conor:
That's really interesting. So your professional experience in the medical sector sort of helped scope the focus of the audit.
Allison:
Claudia researches all her own audit topics, so I've given her that latitude to go and use her experience to find objectives and find areas of improvement within state programs she has her medical experience that I, I don't know. I trust her to go and use her experience to come up with areas where we can improve,
Conor:
Once you decided sort of the focus of the audit, the objective, can you tell us a little bit about how you went about your work actually conducting the audit?
Claudia:
Yes. a lot of research, went into this. So, like I said, these vaccines were new, so I went right to the vaccine manufacturer websites. I looked. You know About the vaccines, how they came, if they were multi-dose vial. Single dose vial, how they were reconstituted, looking at. the process from a nurse lens what they need to do to, get that vaccine to people. and looking at the storage and handling, what kind of came up as a theme was the cold chain. So the cold chain is the documentation that needs to happen from basically the time that the vaccine comes off the manufacturer's line until the time that it's injected into somebody's arm, and so there needs to be a process and documentation ensuring that that vaccine is kept at those optimal temperatures throughout that period of time. The information I got there led me to the CDC. So the CDC put out an overarching vaccine handling and storage playbook. And that was the criteria that we used for this audit. It basically told the warehouse where these vaccines were coming into, the processes they needed to ensure that the vaccines were kept at those optimal temperatures and the documentation requirements along the way.
Yusuf:
You've got a nice one pager at the front that talks about four key things that you found in the audit.
Claudia:
Yes. We had three findings that were related to the vaccine storage and handling. Our first finding had to do with inaccurate inventory and storage. So we did an actual inventory of the vaccine. I went to the warehouse where these vaccines were coming in and being processed and then distributed throughout the state. So we did, an actual count of the vaccines and in that inventory system They had, their counts and then they also had the storage units that these vaccines were kept in. So each vaccine had a specific, way that they were to be kept at different temperatures. So the Johnson and Johnson, for example, needed to be kept at refrigerator temperatures. The Moderna needed to be kept at freezer temperatures, but it could also be kept at refrigerator temperatures for a certain amount of time. And then the Pfizer vaccine, needed to be stored at the ultra cold temperatures. So it was a special, freezer unit that needed to be ordered by the warehouse to store these Pfizer vaccines to be kept at the proper temperatures. That was the research, the pre-research that we did. So knowing, what temperatures they needed to be stored at and where they needed to be stored, led us to that process. and so when we went in, we did the inventory, but we also compared their data to where we actually found the vaccine, which it was not correct when we were there. for example, we found most of the Moderna vaccine in the freezer, but it was listed in the refrigerator site, and so that was a discrepancy that we found right away. And, we keyed in on that.
Yusuf:
There's a last little graphic that you've got in the report that explains the cold chain process.
Claudia:
The cold chain process starts really at the vaccine manufacturer from the time it comes off the line that that temperature documentation needs to start at that process and be followed by that vaccine throughout the entire distribution movement of the vaccine. It needs to be monitored and kept at those proper temperatures. what the vaccine manufacturer says, so in that graphic you can see that it, it starts at the vaccine manufacturer, and then those doses are packaged up into specific lot numbers and expirations, and then those are transported in temperature sensitive packaging. Because this was so brand new, they were developing these temperature transport packaging as they were going through this process too. So I think North Dakota was actually a test site for some of these, temperature packaging. So the vaccine manufacturer shipped in these temperature sensitive packaging to our location as, a test site, I believe to ensure that it was keeping proper temperatures and the monitoring process was taking place during that time. From there it goes to, by plane or whatever large, shipping units. Uh, I think it was semis, they were using planes. A lot of these were being shipped overnight, and so it was time sensitive, temperature sensitive. there had to be a lot of things that went just right to make sure that these vaccines got to where they needed to go within the proper amount of time. from there, it went to our warehouse. So our warehouse location is in Bismarck, North Dakota, which Bismarck is centrally located in the state. And, North Dakota is a large state, so I, I don't know if a lot of people realize just how large we are. So the north east corner of the state Grand Forks is probably three and a half hours drive from Bismarck, and so for North Dakota, we had one distribution site. There were a couple different chain pharmacies that would get, vaccines shipped right to them, but the majority were coming to the state warehouse and then being distributed from there. So those vaccines would come into the state warehouse and we actually went to the warehouse and watched them process, shipments and saw how they, from there, how they recorded the lot numbers and where they went. So from there at the warehouse, they would go into our storage units there, whether it be the ultra cold or freezers or, or fridge depending on the vaccine manufacturer. And then from there, there was a process where, entities around the state would order the vaccine or how much they needed, and then those were dispersed out. And so from the warehouse, we had these digital data loggers. So they're a small, transport unit basically that, monitored the temperatures from the time that it left the warehouse until the time it was delivered. And so these were supposed to track the temperatures, of these vaccines. And the state also developed, processes where they had forms where, you know, documentation was supposed to be taking place as to what lot numbers were going to what location. they had a temperature graph on there, so the drivers and deliverers were supposed to be checking the temperature of the vaccine while it was en route and ensuring that those vaccines were kept at those proper temperatures. And then, it went to the points of care, so whether it be clinics or hospitals, other entities, pharmacies around the state. And then from there, those, local locations would take over the temperature monitoring and storage of the, vials of vaccine until the time they were administered.
Yusuf:
Did you find that a lot of what was happening within that process was newly developed. So I imagine there'd be some, existing processes that we used. The state warehouse would've been in place already, but would there have been a lot new that needed to be developed specifically for, movement and storage of the vaccine?
Claudia:
There were some of these processes in place because, vaccine for Children, I believe they were distributed from the warehouse as well, so they would have to have some of these processes already in place. the Pfizer vaccine, the ultra cold, I don't know, I can't speak for that as to if they had processes for the ultra cold. But, I do know that they would have transport units that would, transport at the freezer temperatures and the, refrigerator temperatures. but it was just the volume. this is when the vaccines were coming out and so everyone wanted them. and the state was trying to get them out to people who wanted them as fast as they could and trying to ensure that, you know, they were kept at these proper temperatures and, and this process was maintained along the way. But yes, the volume of it I think was not something that they were used to or prepared for. they. a good job of preparing and had processes in place. but we found some areas that potentially they could improve upon.
Yusuf:
Okay. And so those. areas where, improvements are possible could help with, ensuring the proper storage and handling of vaccines that are typically administered to children as well. Given that, it used a lot of the existing processes.
Allison:
Yes. Our flu vaccine comes into the state warehouse as well, so they do have procedures in place, but like Claudia said, the volume of the Covid vaccines coming in, you know, to be prepared for that and to be able to take the same process and try and implement it on a brand new vaccine like that with all of the requirements, that's where maybe some improvements need, you know, needed to be evaluated.
Yusuf:
What has feedback from citizens been.
Claudia:
The feedback that we got was, overall positive. People want to know. Vaccines, were, kind of a polarizing subject and, regardless of how you felt about the vaccine, it didn't really matter for this audit. It was, it was looking at the way that the state stored and transported and, moved these vaccines and if they were kept at the proper temperatures, that's what we wanted to keep it at. not political in any sense.
Yusuf:
Would you be looking to do any similar audits or follow up work?
Allison:
Yes, we do follow up on all of our findings this one did take a while to get through. This audit is actually coming up again this year in 2023, so we'll be doing some follow up on this, very soon. and so they've had a short window to make some improvements, so, we'll see how they're doing. And I know they started doing, corrections even during our audit.
Yusuf:
In terms of, the techniques and skills that were used for this audit, in terms of planning, in terms of field work and in terms of reporting, are there any lessons that, you think would be, important for other performance auditors that are conducting similar audits to think about?
Allison:
Communication, always these sensitive areas. making sure that the place that you're auditing and the, the people, they're human, that they understand that we're all on the same playing field. We're all trying to make an improvement here. especially in hindsight, you know, you can look back and see where things could have been improved, but when you're in the heat of it and you've got, so much, you know, going on in these labs and short staffed and during a pandemic to try to get people to come in, and socially distance and trying to implement these processes and, get all this, this shipment out to people. just to be sensitive cuz you're going back in auditing afterwards. So just being sensitive to the situation that happened while it was going on, and. People were also trying to live their lives during that time, not just track vaccines, you know? So just being sensitive to that as well when you're trying to audit those sensitive areas.
Claudia:
it was, really important to stay within the scope of what we were looking at too, we could have went off on a hundred different tangents and a hundred different ways but, staying within our objective and, and within the scope of what we were looking at too.
Yusuf:
What do you see as the next areas that you'll be looking at particularly as it relates to health. And I'm not saying, you know, reveal all of your grand plans to us around health auditing. But given what you've seen bringing in specialist skills into this audit, where do you think you'll be going next.
Allison:
Our Department of Health and Human Services, they have so many health related areas.
Claudia:
Vaccine for Children Program, there is Women Infant and Children program. there are a lot of health programs around the state I'm excited.
Allison:
Some others might be even like developmental disability, there's just so many areas that we have not looked at, haven't had a chance to look at as well.
Conor:
So these are all important health areas that impact every person in the state. How do you go about selecting, cuz you can't do them all, right? How do you go about selecting which one is highest priority? Or like how do you put them on the plan?
Allison:
Public impact ones that'll have public concern to reach a large audience.
Claudia:
Risk areas. Yeah, some of the higher risk areas of health, which I think is why COVID was so important to a lot of people just because of the nature of what was going on.
Allison:
Working with the places that we audit to and ask them where can the audit be helpful. We want our audit to be helpful. And so, for us to look at an area that they're already looking at or was just revised, that type of thing. We also want to provide information for them and find ways that we can help them improve. So we do look for feedback from them. Of course, we're the auditors, so, they don't always want us to look at certain areas either. But we do try to take away that stigma try to, be all in the same playing field and if there's an area for us to look at and help with, media impact or legislative impact. Cause those are the types of things that get results.
Conor:
Just sticking with legislative impact, so, recommendations go to the audited agency. And then your legislature also looks at your reports. Do they tell the agency to implement the recommendations? Or is it more advisory or, how do you make the change that you've found needs to happen in your reports?.
Allison:
We are seeing more change happen more quickly. More requests for our audits be made from the public and legislative changes coming as a direct result, while legislative changes can come, even just in committee, calling the agency in and asking how things are going, what they might need, as well as our legislature, making law changes that might require, certain action to happen. Also considering agency's appropriation. if it's an area that the legislature wants to further invest in to expand or help that agency to implement improvements. Yeah. As I mentioned before, the we have increased use of the media and releasing and discussing our reports and that media attention can, be uncomfortable for some of the entities that we audit, especially if we're going back and we're seeing that findings haven't been implemented for several audits. And so we are seeing change as a result of that as well, even starting during the audit so that when our report does come out, they can say right away that they've already started changes. And so then the citizens, can see the work that we're doing in the news and they can, see that they have an agency that they can reach out to that can look into some of the concerns that they might have as well.
Conor:
Three key takeaways from the discussion today for me, number one, understanding what matters most to your citizens. So with the vaccines audit you did in the report, there were really significant consequences if vaccines weren't managed and stored properly. Number two, the real value of having, different skills and backgrounds in performance audits. So, you guys were lucky to have your skills, Claudia, with, nursing and medical background that really was able to refine and shape the scope of the audit. And thirdly, the performance improvement that we talked about. So you looked at vaccine storage and handling, but really there are broader lessons there for improvement in how other vaccines are handled. So, three things that I learned today.
Yusuf:
I'll add one and say that the, way in which the report is structured is really easy to read for a variety of audiences and particularly for, those individuals that may not want to read hundred page reports, which is probably 99.95% of the population of the globe. It's really great to see short infographics, nice, easy to understand process flows, and not walls of text.
Claudia:
Yes, Callie and Emily, our communications team, are the ones responsible for those infographics and putting this report together. So, an average citizen, average North Dakotan can read this report, and that's, really important for our office. We want this information in their hands as well.
Conor:
That engagement's really important. So how can listeners and North Dakotans, how can they get involved in your work?
Allison:
On our website, you can sign up for our media releases, when our audits come out. We do have a Facebook and LinkedIn page that the public can join. On our website you can find where you can submit audit topics, ask audit related questions. Our state auditor on Wednesday mornings is on the radio, talking about our audits from 10 to 11 central time. It's at whatsonyourmindradio.com. If you wanna listen to our state auditor. But most importantly, just being involved in your government. Follow the legislative processes and communicate with your representatives on where improvements are needed. Know who your elected officials are and how to contact them.
Yusuf:
We'll put links to those into the show notes. Ladies, thank you for joining us today on The Performance Audit Report and giving us a detailed explanation of the recent audit that you did, and the office overall.
