Maura Hayes-Chaffe is the Deputy Comptroller for Audit in New York City.
Maura joined us to discuss the Bureau of Audit, including:
Conor:
Hello and welcome to The Performance Audit Report. My name's Conor McGarrity.
Yusuf:
And I'm Yusuf Moolla.
Conor:
Today we're very excited to have a special guest with us from the New York City Comptroller's Office. Maura Hayes-Chaffe. Maura, welcome to the show.
Maura:
Good evening. Thank you so much.
Conor:
Could we just kick off briefly with your background and your role there at the Comptroller's office
Maura:
Sure. So I'm known as the Deputy Comptroller of the Bureau of Audit, within the Office of the City Comptroller, and our principal function is to conduct performance audits of agencies, vendors, and other recipients of New York City funding.
Conor:
New York's obviously one of the great global cities. Can you just give us some idea of the scale of public services in New York?
Maura:
I'd start by saying that there are about 8 million residents living in New York City. They're spread across five boroughs. The boroughs are very different from each other, and they have very different demographics very different economic profiles and so forth. There are about a hundred agencies that are providing services to New Yorkers, and in total, the budget is about a hundred billion dollars a year.
Conor:
Huge enterprise there. Given all the work that you have to do to audit those agencies, one of the things we've heard from other guests has been the difficulty in recruiting and attracting performance auditors and other audit staff. Is that a challenge that you faced also?
Maura:
Yeah, it is absolutely a challenge. The challenge really comes about, in part because we're a civil service structure, and so it's essentially exam driven, you have to pass certain exams before you can even apply. And so the pool is already small. And then beyond that, we struggle with the fact that salary scales are not competitive with private sector. And I think on top of that, New York City, like many other places, Has undergone enormous change as a result of the three year COVID emergency, right? It's changed people's expectations. It's changed their desires. I do think that we have a couple of advantages. One is that the work itself is very interesting. The other is that the office of the city comptroller allows a hybrid schedule, so we're actually allowed to work two days a week from home, which is very unusual within New York City government. Most agencies at the present time are required to work in person five days a week. So we do feel It's an advantage, but perhaps, we'd be happier if there were civil service changes in salary and so forth that made it possible to attract greater experience to begin with and also to retain people over the long term. That said, I will say we have a number of audit staff who've worked with the Bureau for more than 20 years. One of my direct reports has been with the agency for over 40. She started as a trainee and is at this point, Assistant Comptroller of the Fiscal Operation.
Conor:
So you mentioned they're the changing expectations of auditors and people maybe starting their careers. Have you seen much in the way of the changing role of performance auditing over the years or maybe innovative delivery approaches in your work?
Maura:
I think the Office of the City Comptroller, you know, this administration started a year and a month ago, and we are definitely in process of implementing significant change and new approaches to performance auditing. Part of that is attitudinal you know, we would like to approach auditing from the standpoint of the fact that the auditees, the agencies being audited, have more alignment with our shared values. We have shared values even though we are technically an oversight. At the end of the day, both the Auditee and the Office of the City Comptroller would like New York City to function better. For residents to be happier with the services that are provided and that there is fiscal responsibility across the board. So whilst nobody likes to be audited and some of the agencies may be skeptical, we are approaching this from the standpoint of, Hey, please work with us. We would like it if you are happier with our findings and with our recommendations. And we would like it if the recommendations were implemented at the end of the day, right. For an audit, a particular performance audit, to have an impact, both parties have to agree the change is needed and for change to be implemented. So that's a new approach for this administration.
Conor:
You mentioned there that a great deal of it is to do with attitudinal change, both on the side of the auditor and the auditee. Have you seen any change in that respect, brought on by COVID? Are agencies more willing or less willing to engage and to implement recommendations?
Maura:
I don't think so much that it's altered attitude in that respect. I do think it's altered the environment in which we're auditing, you've got COVID, we're coming out of it. But there are also economic impacts. New York City is facing, I would say uncertain budgetary times and there are a lot of vacancies, not just at one agency, but at multiple agencies. So when we come in and we say to them, for example, you're not doing enough, a logical response is yes, but we don't have the resources to do it. So one of the things I say to my staff is, you know, you need to have that conversation. Don't just ignore a response that says we don't have the resources. Engage around whether or not there's an alternate way to achieve change that is achievable within the resources that are there. We're also going to end up in some cases saying, Hey, resources apart, you really do need to do better. So it's a work in progress. At the end of the day, agencies do need to do better in some cases, and if that means they need to work harder to get more resources, then so be it. it's a difficult thing. The other major change here, I would say is a huge interest in equity. Auditing for equity and also in a related way, focusing on participation by stakeholders.
Yusuf:
Can I just ask a follow up to that? So where you have, agencies that don't have enough resources to be able to do everything that they need to do to be able to achieve their mission fully. Do we play any role in helping them prioritize activities so that they can focus on the bigger ticket items?
Maura:
That's an interesting question. I think the current comptroller does play a role in setting policy and attempting to drive policy, not necessarily through audit or within the audit shop. But he is a very active elected member, who works very closely on budget issues, for example, and advocates for change that will for support agencies. Now, in the context of individual audits, it really depends to a large extent on the scope of the audit and what comes up. But we do, as I said, try and engage with agencies and we are training staff to say, okay, you don't have resources to do that. Is there something else you could do? And we will modify our recommendations to a certain extent to allow that. Now, again, the extent to which we can do that is going to depend on other things. For example, legal requirements. If there are legal requirements and legal mandates that an agency must meet, then they just have to meet them. And an audit is going to say you have to meet them. We may advocate outside of the audit through other bureaus and through the comptroller himself to try and get more resources. But at the end of the day, the audit is still going to recommend that those mandates be met.
Conor:
You touched there on the growing theme of equity in the work of performance auditors, and it's certainly, something we've observed more broadly, and you mentioned specifically the term people powered audits. I think. Was what you called it. Can you, can you talk to us a little bit about that please?
Maura:
Sure, when we are talking about people powered audits, we're talking about engaging the community and other stakeholders and asking them, a range of things. For example, if we can engage them pre-audit, what would you like us to audit? What issues are important to you? What answers are you seeking? Where do you feel there's less transparency? And you would like greater transparency? Where can we shine the light that an audit report offers to suit your needs, right? And that's driven in part by a recognition that the end users of services are in a better position than anybody else to identify some of the risks and some of the issues and whether or not an agency is performing as they should. So we want to engage them and ask what would you like to audit? Now, we do that in a range of ways, and some of those ways have been ongoing for a very long time. For example, our website has a suggest an audit link. We also have a hotline, so those sorts of things have been up and running for a very long time, but we're also engaged in our process with the New York City Housing Authority, for example. The New York City Housing Authority Project or pilot, if you like, a people powered project, started way back in the summer and it was designed to really engage with residents. It is the largest public housing authority in the United States and we wanted to ask folks, how is NYCHA doing? what are NYCHA issues? So we embarked on this process of doing so. This is well before an audit started. It wasn't driven entirely by the audit bureau. There are other bureaus in the Office of the City Comptroller who were really driving that. Our policy arm, our public affairs arm, and we set up a series of in-person interactions to ask. Right. What are some of the issues you face? We also built into that some level of education. This is what an audit could achieve for you. These are the sorts of questions an audit could answer. And we also surveyed people. we collected over 700 responses. They were in multiple languages. They were spread across the boroughs, spread demographically to residents to ask, what are some of the issues that you find and that would be useful for us to audit. So one of the really striking things about that exercise, it went on roughly from June, July to January, we had incredible consistency across the board in person and in writing. There were a set of issues that regardless of the location of the development, netted the same things. We've got concerns about security. We have concerns about repairs and maintenance. We are not convinced that NYCHA is spending money wisely. In the end, we took all of that in. We looked at it. The audit team put together about 15 ideas that were on point, and then we sat down with an established resident audit committee and asked them to vote. They selected two audits. It was, again, very quick, very clear. There was very little debate. They coalesced around two issues. One was repairs and maintenance as it relates to vendors, how does NYCHA hire vendors? Does NYCHA do a good job overseeing vendors? How do they make sure that the work is completed by vendors? How satisfied are residents with the vendor work that's been completed, right? That was one. The other one that came up relates to a rental assistance demonstration project known in New York City as RAD. Which is being used and piloted to convert some of the usual housing stock to a different federal program. Now, from a city perspective, it was designed to open up another source of federal funding. For NYCHA residents, and for the developments. The developments are very old. It's fair to say that upkeep has not kept pace over time. And so there's budgetary issues that prevent, the amount of repair and upgrade that's necessary. So this city has latched onto this as a possible means to a solution to increase funding. But because it's a different funding stream and a different program, it's voucher driven. There's a certain skepticism by residents. Residents have angst about this. They're not convinced that it is good for residents. And so the other idea that was selected was an audit of the eviction rates in the RAD housing versus the general NYCHA public housing. So we've started both. And over the course of the sitting of the committee, which we expect to be about a year, we will at periodic intervals go back to them and say, this is some of the things we are finding. What do you think? Is that consistent with your own experience? Do you think we're on the right track? Are there other things we should look at and so forth? And we will continually do that. Now, we will do that while we are maintaining compliance with GAGAS, which stands for Generally Accepted Government Auditing S tandards. It's a federal standards, a standard we're required to follow based on the New York City charter. And that will mean that the principle thing is that we need to maintain independence. So whilst we are taking information from residents and gathering input, it will only be one input. There will be other inputs, and we will have to weigh them all objectively to reach evidence-based conclusions. But it's a very interesting thing. I don't believe many municipalities across the United States are actually engaging in this type of work. And the other, you know, during the course of the NYCHA, we were very conscious of the fact that while we're meeting with people over the summer, We really were thinking we weren't gonna have final audit reports out until a year from now. That's a very long time to expect community members to stay engaged. So we also developed non GAGAS approach called an observations, intended to address security concerns. And we went out. Now the bureau has between 130 and 160 people, depending on what our vacancy rate is looking like. But at the time, we mobilized a hundred auditors who went out into the field simultaneously and inspected every single door in every single NYCHA development in the five boroughs. It took us about a week to do that in person, and we followed a checklist and we found that more than 60% of the residential buildings had doors that did not work. They didn't close. They had broken windows and so forth, so that was intended to address security, right? If your front door doesn't work. If you can't secure your building, you have very little security as a building. right? NYCHA's expected to do that. NYCHA's responsibility as the oversight agency and also the landlord is expected to maintain things at a very basic level and they have sadly fallen down. Right. It echoed, it was a follow-up observations earlier on, it had been done several years prior. It took them, well, like eight weeks because they didn't mobilize the entire staff. And we found that this time when we went out, conditions were markedly worse, which was really sad to see.
Conor:
You've got many agencies that you can potentially audit or currently audit, how did you decide on NYCHA as a candidate instead of other agencies for this people powered approach?
Maura:
There are a couple of things. One is that public housing and low income housing is a really important issue in New York City. NYCHA represents a massive amount of the low income housing available in the city, so it's really important that NYCHA succeed. That's one thing. The other is that NYCHA has had many years of difficult issues, and the feeling was that we wanted to give NYCHA residents a voice. When we talk about people powered audits, I spoke about equity. That's one thing. Two is, we know People who actually experience an agency have better insight into the risks that are present. The third thing is that we want to help residents make change, right? The hope is that if we engage through an audit process, and if audit reports and audit results are fully supported by community stakeholders, then there's a greater chance of change. We don't have the authority to say, NYCHA, you must fix X. We don't. We recommend it. We hope they do. But in general, audits that gain a public, that result in a groundswell, are more likely to lead to actual change. We know that works for democracy too, right? If voters are engaged, you have better outcomes, so it makes sense from an audit perspective. If we engage, the community, not just the residents, but other interested community groups that we can help leverage change. So part of this is the audit reports itself, but I spoke of the pre-audit engagement and there will also be a post-audit engagement. So that won't necessarily involve the Audit Bureau, but it will involve the Office of the City Comptroller, partnering with other community stakeholders to really lobby for change. And the hope is that it will actually help drive change one and two, that NYCHA residents will feel that they did have a seat at the table, that they were heard, that they do see and hear their voices echoed in the audit results. And that's the hope here, you know, from a methodological point of view, NYCHA residents offer a group of people that are readily identifiable who all share the same level of interaction with an agency. So it was a very good pilot. There will be others. We've recently developed a participatory framework in conjunction with a Canadian consultant, CAAF, the Canadian Auditing Accountability Foundation, we engage them to help us develop a broader framework that could be applied across the board. So we've only had that final document produced for us. We're in process of implementing it fully, but there are a number of areas in which we can see that it would make sense for us to engage. And to probe how New York City residents feel about certain services and also what they need, and to allow those things to drive our decisions about who we audit when, and also to engage with them throughout the process.
Conor:
So that NYCHA audit and ongoing works has had significant and will continue to have significant impact. Another one of your audits that seemed to have resonated a lot with citizens in the community was your review of the Economic Development Corporation and the New York Ferries. Can you just tell us a little bit about that?
Maura:
Sure. So the EDC or the New York City Economic Development Corporation is only a semi agency in the sense that it doesn't function the way other New York City agencies function, but it is the recipient of vast amounts of money for the purposes of helping to develop New York City. So they were asked under the prior mayoral administration, Bill De Blasio, to expand the ferry system. You know, New York City sits on many miles of, riverfront. There's the Hudson River, there's the East River, and There are quite a few transportation deserts where people are not within easy access to a subway and whatnot. So it made sense for the ferry system to be developed, but when the ferry system was expanded, it was expanded through the selection of a vendor, and it was predicated on a certain level of funding and with a certain level of subsidy by New York City government. At that time, the city subsidy was predicted to be about $6.60 cents per head. Well, during the course of the audit, we found that reported subsidies and the actual subsidies were really not accurate. They had reported in some years, for example, a subsidy of about $8 a head. We found that in 2021, for example, the subsidy was close to $13 per head, so that was a significant finding. We also found that, and it was tied to that, a lot of the expenditure had not been reported fully. So there are about $243 million in ferry related expenditures that were not identified as such in the financial reports of the EDC. So the audit shone a light on that. We also found in our opinion that there were a number of ways in which money had not been spent wisely. The tune of something like $60 million that either should have been recouped or should not have been spent. Of course EDC did not entirely agree with our findings. That's fairly typical in audit work. And their response is also attached as an addendum to the published report. But I will say the EDC fer ry audit is very important to us for a couple of reasons. One is a really good example of how transparency can be expanded by an audit, right? It's one of the core functions of a good performance audit, making sure people really know what's going on, right. The second thing is that it led to change almost immediately. Unlike Sydney, where the ferry fare is dynamic and based on distance and demand and things of that sort, New York City had a flat fare. No matter how far you were going, no matter where you were going, no matter what your income level was, no matter whether you are a tourist or a resident, flat rate. So almost immediately on the heels of the audit report being published, the current administration announced changes. they lowered the rate for certain folks with lower incomes, and it also boosted the rate on certain routes to a higher level. The other great change that came about was that, EDC committed to launching a new request for proposal process to seek another vendor. Now, it may not be another vendor, but at least they went out. The RFP has now been published and they're in process of reviewing to see whether or not the current vendor, who is Hornblower should remain the vendor. And so I think that's, those are two really instant results that we are very proud of.
Conor:
you spoke there Maura about the importance of transparency and how providing that transparency can lead to meaningful change and you give us the example there of, new fare system coming in One of the transparency initiatives we've noticed on your website is your recommendations tracker, where you track recommendations made. Can you just tell us what that's about and the rationale and what that's gonna achieve?
Maura:
We wanted to put up a public facing recommendations tracker because, Whether or not recommendations are implemented is really crucial to the question of whether or not there's been actual change. In the prior administration, recommendations went out and there was very limited follow up. There was a kind of once and done approach. We wanted to be able to track more systematically the results of our audits. So beginning with audits that were issued by this administration in January 1st, 2022, we began tracking and collecting information from the agencies more regularly. We're doing it roughly three to four times a year, and we'll continue to ask them, where are you with your recommendations? Until they say, yes, it's been implemented, or no, definitively we are not going to implement it. But that's all present on the recommendations tracker so that if you have a particular interest in a particular audit, you can go on at any point that you desire and just look and see where are the recommendations? Did the agency in fact implement what they promised to and has there been a change of heart? incidentally, we do find often that agencies at the time of the audit response said, no, we're not implementing that, will in fact have implemented it when we go back a year later and say, yeah, we did actually implement. They maybe didn't want to, but they reluctantly agreed that it should be done, and they've in fact done it.
Yusuf:
A lot of our listeners are interested in how different audit offices use data in particular. So some audit offices have been using data for a very long time. Just wanted to understand what within your audit office the use of data looks like. Do you have dedicated resources and where is that going in terms of being able to help support your audits?
Maura:
I would say data availability falls into two buckets. Data that we have available regardless of conducting an audit and data that we obtain during the course of an audit. I think there has always been, or consistent use of data of the second bucket right? We've gone to an agency, we're conducting an audit in this scope, we've asked for data within a particular framework, and the audit teams have become very good at analyzing that and using it to determine where risks are and to also determine what audit tests should be conducted. What's relatively new is a data driven approach to selecting audits themselves. We've recently added resources in our team and also across the Office of the New York City Comptroller. I think there's broad recognition by this administration that data is, in this day and age, instrumental to all sorts of decisions, including where you should audit. So I don't know if you are familiar with it, but New York City has a good deal of data that is publicly available on the open data website. So we have recently begun data mining within it, for example, to look at mayor's reports, right? A lot of agencies report publicly on their KPIs. How are they doing? So we've started mining for trends to see are things going up, are they coming down? For example, there's also complaint data there that gets reported. So we've started data mining around complaints.. There's a lot of other things. We also have access to a good deal of revenue and expenditure data by virtue of the fact that we are the comptroller's office. And so we are also data mining there. We're looking for trends, we're looking for outliers and we're chasing them. So, you know, it's relatively new for this office, but I feel very strongly that's an important way to go. is best practice for any fraud, waste, abuse operation, and so we are ramping up in this space to ensure that we are really doing all that can be done.
Yusuf:
So we have come across the New York Open data portal. Really useful information in there. I think a lot of cities can learn from what New York has done. I'm sure there's, a lot that you find that can be improved. But in terms of, the overall setup the ease of access to the metadata. I remember once even asking the people who are responsible for the open data portal, we'd like to use your data in a demonstration. And they came back and said, no problem, go for it. You don't even need to ask us if you want to use our data. So really great to see.
Maura:
This office also hosts every year a whole data week in which we teach members of the public and other agencies, Hey, this is all of the available data. This is how you can access it, and this is what you can find there. If you look at our website, you'll also find that this administration has done some amazing things in standing up dashboards, so that data is readily available to people in a way that makes sense to them so they can actually see it. Our office is incredibly proud of the climate dashboard, which uses data to show how New York City is doing, not just across the board, but by neighborhood. You know, if you're interested, you can go and look. How's your neighborhood doing in terms of emissions or other markers of climate work? How's New York City doing as a whole? What are some of the driving concerns, right? What's driving emissions and so forth? It's all available there in a very user friendly way, and so we're proud of that. You know, the other piece of open data that this office created a long time ago and it continues to make available to folks is Checkbook. If New York City wrote a check or dispensed money to someone in check form, you can track and you can data mine the expenditures, by payee online. They're pretty cool.
Yusuf:
It is cool. It is cool.
Conor:
Great conversation. Maura. A few lessons that I've learned. There are certainly a few key takeaways for me was the importance of engaging citizens early on in an audit and getting their ideas as the potential scope of particular audits, and especially those vulnerable cohorts or those where the audit's gonna give voice as you said to those people. The other thing was the importance of when you're engaging citizens, you need to reduce the audit timeframe because they may not be as engaged in one year's time as they were at the outset, giving you their ideas. So really handy tip there. And the other key takeaway was, The use of open data where it's available by comptrollers and state auditors and auditors general, and are we really doing enough as a profession at the minute to actually use what open data is currently available in all of our jurisdictions? So three fantastic things there for me.
Yusuf:
So I'll add one to that the way Maura you explained the NYCHA process with, going back through the year to committees to say, are we on the mark or not? That sounds like a fairly agile process, which makes a lot of sense. So it's not just static and this is our scope and this is all we are ever gonna do, and then produce a report at the end. So some flexibility to accommodate what stakeholders want, while still maintaining independence and ensuring that the standards are kept. So that I found quite interesting.
Maura:
Great. Thank you.
Conor:
Okay, so just wrapping up some fantastic resources, dashboards documents on your website. So I'd encourage all our listeners to check those out. But more broadly if auditors out there want to get in touch with you, what's the best way?
Maura:
I'd be happy to share my email address. It's MHayes1@comptroller.newyorkcity.gov. I'm also present on LinkedIn, and you will see me on the website.
Conor:
Great conversation today, Maura. Really appreciate your time.
Maura:
Yeah, I really appreciate the invitation. I think it's extraordinary that there's a podcast on performance auditing. Who knew? So good on you both for taking this on.
