Fraud Prevention: Data and Analytics in Law Enforcement

How does law enforcement use data to prevent fraud? Kelly Tshibaka, Chief Data Officer of the U.S. Postal Service Office of Inspector General, and Caryl Brzymialkiewicz, Assistant Inspector General & Chief Data Officer at U.S. Department of Health and Human Services Office of Inspector General, speak with CXOTalk about how different agencies at the Office of Inspector General use data analytics.


Sep 08, 2017

How does law enforcement use data to prevent fraud? Kelly Tshibaka, Chief Data Officer of the U.S. Postal Service Office of Inspector General, and Caryl Brzymialkiewicz, Assistant Inspector General & Chief Data Officer at U.S. Department of Health and Human Services Office of Inspector General, speak with CXOTalk about how different agencies at the Office of Inspector General use data analytics.

Since 2015, Tshibaka has served as Chief Data Officer in the U.S. Postal Service Office of Inspector General (OIG), where data analytics has resulted in more than $920 million in financial impact or audit findings in FY16. She previously served as the Acting Inspector General of the Federal Trade Commission, worked in the Office of the Director for National Intelligence (ODNI), and served as the Special Assistant to the Department of Justice Inspector General.

As the first Chief Data Officer for the OIG within the Department of Health and Human Services (HHS), Brzymialkiewicz is focused on providing more and better access to data and analytics; accelerate analytics for use in audits, investigations, evaluations, and compliance oversight; and enhance OIG’s use of data to make more informed decisions. She previously served as the Deputy Assistant Secretary for Risk and Decision Analysis at the Department of Homeland Security (DHS). She also previously served as the Operations Research Division Chief at the Joint Improvised Explosive Device Defeat Organization, and led a team of analysts at the Center for Naval Analyses.


Michael Krigsman: Data, analytics; it is everywhere! And, today on Episode #253 of CxOTalk, we’re being visited by the feds! No, that’s true! We are. We have two amazing women who are going to be talking about the role of data in law enforcement and these folks are fantastic and they are right in the middle of it.

I’m Michael Krigsman, an industry analyst and the host of CxOTalk. Before we start, I want to say “thank you,” as I always do, to Livestream because those guys are the best. They provide our video streaming infrastructure and if you go to, they will give you a discount on their plans. And so, thank you so much to Livestream.

There is a tweet chat going on right now using the hashtag #cxotalk. And, if you join in on Twitter using #cxotalk, you can ask questions and you can participate in this conversation. And, if you’re on Facebook, then go over to Twitter, because that’s where the chat is happening.

So, without further ado, let me introduce the first of our two guests, Kelly Tshibaka is in the house and hey, Kelly, how are you?

Kelly Tshibaka: I’m good! Thanks for having me today!

Michael Krigsman: So, Kelly, who do you work for and what do you do?

Kelly Tshibaka: Currently, I work for the US Postal Service, Office of the Inspector General. I’m the Chief Data Officer here. I’ve been here about two years. Prior to this, I’ve mostly done my career in the Inspector General Community. I was at the Department of Justice, the Director of National Intelligence, and the Federal Trade Commission OIGs. I’ve done audits, investigations, inspections, legal counsel work, congressional relations, so getting to do the data analytics piece is just adding another element to the layer of oversight and fighting fraud.

Michael Krigsman: Okay! Well, we’re going to be learning a lot more about this during the next forty-five minutes. And, our second guest is Caryl Brzymialkiewicz, who is Kelly’s peer, I guess we could say, in another agency. And Caryl Brzymialkiewiecz… You know, there’s no words and no excuse. Caryl Brzymialkiewicz, how are you and welcome to CxOTalk!

Caryl Brzymialkiewicz: Thank you so much, Michael! Hey, you got Brzymialkiewiecz right, so I give you that! If you change Kelly and my name around, that’s alright! You got our last names right, so congratulations to you for that.

Yes, I am Kelly’s peer. I am the Chief Data Officer at Health and Human Services, our office of Inspector General. I’ve been here a little over two years as well. We are the two Chief Data Officers in the Office of Inspector General community, so it was exciting to meet Kelly and to see what she’s trying to do in her organization and a lot of what we’re trying to do in our organization. So, we’re happy to share what we’re doing from our piece. You know, you mentioned the Feds are here! So, our small piece of what we’re trying to do; happy to talk about that today!

Michael Krigsman: Fantastic! And, let me ask either one of you to explain for us… Maybe, Kelly, you can explain what is the Office of the Inspector General and how does it fit? You work for the post office, Caryl works for Health and Human Services so how do the pieces fit together?

Kelly Tshibaka: I’m so glad you asked! I actually really love this community. When I first interviewed for my job at DoJ, I had no idea what an Inspector General’s Office is. And then, I had come to find out that I actually really love it. We’re essentially the law enforcement internal affairs people in the federal agency. So, nearly every federal agency has an Inspector General office, and we do audits and investigations for the purpose of detecting and deterring fraud and abuse, and promoting the efficiency and effectiveness of the government agency. That’s basically the legal language for saying, we make government better. We’re the people assigned to that, and yes, we have job security because we have an impossible job.

Michael Krigsman: So, you report, then, both to the agency as well as to the central Inspector General office. Would that be the right way to say it?

Kelly Tshibaka: No.

Michael Krigsman: Okay.

Kelly Tshibaka: Although, I appreciate the attempt. There is not yet a central Inspector General. Congress has considered it. We have dual-reporting to our agency head and then, actually directly to Congress. And so, our oversight committees are Congressional Senators and Representatives who are interested in our work. Those are the people who we are accountable to, and that’s how we preserve our independence, by having two separate people who we have to report to on the progress of the agency that it is making, and where it’s having its management challenges and how we can help it. That’s how the Inspector General does its job.

Michael Krigsman: And, Caryl, can you maybe elaborate on this, because I think it’s a very interesting point. So, you report to two separate people in order to ensure that as an inspector general, or working in the office of the inspector general, that you maintain independence. That’s very important. Could you elaborate on that point, please?

Caryl Brzymialkiewicz: Yes. Absolutely. So, for our organization, for example, health and human services is a trillion-dollar portfolio. So, our responsibility is to provide oversight for the agency. Think about centers for Medicaid, Medicare, think about the Center for Disease Control, the Food and Drug Administration, the National Institutes of Health, and many, many other agencies. But, that duel-reporting structure really allows for, as Kelly mentioned, that the independence… We want to give the information, first and foremost, to the agency to help make the agency better but we also know that there’s a lot of things that Congress is expecting us to make sure we’re using the dollars as most effectively as we can.

One thing I will say, too, that’s interesting about the law enforcement angle that you mentioned, we’re not just trying to knock on doors within the agency and figure out where people may or may not be filing the right paperwork, if you will. Really, what we’re trying to do is protect the integrity of the programs to make sure that the beneficiaries of those programs are getting what they need, and to make sure that outside entities aren’t… You know, when we think about fraud, the outside entities aren’t taking advantage of the programs as well. So, it’s not just an internal focus, it’s not supposed to be a “Gotcha!” moment.

You know, one of the first things that our leadership team does, especially in this new administration and new leaders are saying… All right, we're an independent source. If you have concerns about fraud, waste, and abuse, come talk to us. We will be objective. We have a lot of subject-matter expertise in the organization to give you an objective opinion. We can tell you where programs might need to be strengthened and talk about those management challenges with the organization. But, they've also engaged in a dialogue to ask, you know, "What's keeping you up at night? Where can we help? Where do we need to look?" Because again, it's really about protecting the programs and protecting the beneficiaries, the health, and welfare of the people who those programs are intended to serve.

Michael Krigsman: And you’re both involved with very high-profile investigations so maybe, can you kind of share with us something about those investigations and then, let’s drill down into the data aspect of both Chief Data Officers. And so, let’s talk about that. Kelly, can I ask you to either talk a little louder or move a little bit closer to the microphone?

Kelly Tshibaka: Oh, sure! Not a problem! Is that better?

Michael Krigsman: That is definitely better.

Kelly Tshibaka: Okay! I’m really excited about things that we’re working on that my Inspector General testified about yesterday, actually, before Congress. So, using data analytics, a couple weeks ago, we started exploring a new dataset that we’re receiving from the Postal Service. The Postal Service is getting more and more advance-electronic information; you know, national postal agencies, on the packages that are inbound to the US. And so, under their direction, our IT’s direction, we started looking at the dataset. We’re using data analytics to see if it was possible to look for trends, or patterns that could identify suspicious parcels that are coming in that could contain narcotics, particularly opioids like fentanyl. For the viewers who don’t know, fentanyl is a highly powerful narcotic that is up to a hundred times more powerful than morphine, and it is oftentimes lethal even upon contact. So, it poses a real danger to the postal service employees who are working and handling those packages and not knowing that they’re coming in.

It turns out that the postal service receives about two hundred and seventy-five million packages from overseas every year. And, the CDP and the Postal Inspection Service is working to try and identify the packages that contain opioids. Well, we ended up… We happen to be invited into a case where it appeared that a postal service carrier was colluding with drug traffickers from China in order to deliver opioids here in the US. And so, we were working that case and when we made the test and confirmed that the package contained fentanyl, we were able just to trace back that sender with the data and say who else has that sender sent packages to in the last several months?

And then, we looked at those recipients’ addresses to identify other people they were receiving packages from internationally and found some overlap where it looked like shipper one and shipper two shared some customers in common, and it turns out we confirmed with CDP that both shippers had several packages seized by CDP and that they contain fentanyl. So, with one case, and going through data analytics, we were able to identify over twenty-hundred suspicious parcels that have come into the US in the last couple months.

Now, what we do with this from here requires a lot of inter-agency collaboration to do something with it. But, you can imagine the potential is just endless for what we can do, using data analytics to try to fight the opioid crisis that our nation is facing.

Michael Krigsman: So, tell us about the data and Caryl, jump in as well. What kind of data do you look at? Where do you get the data? How do you analyze the data?

Caryl Brzymialkiewicz: So, I was going to follow on, too and say that for the opioid crisis in this country, we are also taking a look at that. That's a huge priority for not only Health and Human Services but also on our specific Office of Inspector General. We're looking for how to reduce prescription drug abuse that […] is impacting beneficiaries. And so, part of our takedown into why I actually target medical professionals that were potentially facilitating abuse of these drugs…. And what that really means when you’re talking about it from a data perspective, right? There are a lot of practitioners; a lot of physicians and just medical professionals as a whole that are doing absolutely the right thing. And, what you’re trying to do is if you’re developing, targeting, or sometimes our agents have a question, they need information very quickly to determine whether or not what they’re seeing is behavior that they expect to see, or whether it’s kind of outlier behavior or something out of the norm.

What we’re trying to do is look at the data to very quickly sift through that, give more information to our agents so they can make that informed choice. We’ve created some tools based on CMS’s data environment, so Centers for Medicaid and Medicare have an integrated data repository. It’s a wealth of information and it’s over a petabyte’s worth of information. And, a lot of it’s looking at eligibility, data, enrollment data, provider data, and claims data. And so, when you’re trying to sift through that very quickly, it’s understanding, “Okay, what were the trends in an area that a provider is prescribing something that’s, you know, one hundred times more than anyone else in the local area.” It doesn’t necessarily mean that it’s wrong, it just means we need to understand that and we need to understand that two, is it tied to cancer patients? If it’s something that’s not meant to be pain… related to high levels of pain medications that you would expect, it just opens up a question.

So, concurrent with our takedown back in July with our agents. There were 412 defendants across all areas where there was a specific portion of that which was targeted towards opioid high prescribers. We actually had partnered with our evaluators and released a data brief. It’s called the “Part D Data Brief.” It’s on our website if you’re interested in it, but it was looking at those cases where people are being prescribed extremely high doses of opioids. You know, almost a hundred times more than the CDC recommends. And so, that’s just concerning. It was one of those things that out of about 18,000 providers that prescribe opioids to Medicare beneficiaries, only about 400 were really prescribing high levels of dosages to patients. But, that was about 90,000 beneficiaries.

So, that’s a concern to us. So, when we talk about the data, again, it’s how do you sift through a petabyte; really the Big Data problem; to very quickly query information so you can focus in on a specific provider interest, or agents [can] more quickly give information that they need to help build their case. Likewise, we’re also using some of the data analytics to generate risk models, kind of in a predictive sense, right? And it’s based on good training data. We’re seeing what outcomes are coming from some of our law enforcement cases, and we use that to build several models. So, it’s kind of an iterative. We talk about it being both proactive analytics, and kind of the reactive analytics, but it’s mostly in partnership with our agents.

Michael Krigsman: So, how do you sift through a petabyte of data in order to find that needle in the haystack?

Caryl Brzymialkiewicz: So, I talk a lot about I'm so fortunate to have the team that I do. I think of them as unicorns, those where data talent experts that just have programming knowledge have a statistical background, have subject-matter expertise now, right? The Medicare, Medicaid programs are not easy to understand. And, if you think about your parents, your grandparents, looking at their explanation of benefits from the Medicare programs, [you] usually end up with more questions than anything else. So, all of those medical codes, to understand how to slice and dice the data… And in order to do that, we've got several tools that we use to through that using technology and we're also looking at open source. We've got several commercial products that we're leveraging, and we're leveraging CMS's environment. But, I'll tell you that my folks are just multilingual in programming to be able to come through that.

It’s some tools so that people can get our agents, can just put in… It’s a national provider index, an NPI number that was created several years ago that can put in the NPI quickly into the computer within a few minutes. Our team has already written all the code so it queries the information, and what they get back is a PDF summarized information of everything that they probably like to know. But that takes time, that takes resources to develop the tools. I think what we realize is we kept getting the same kind of analytic questions, right? How much is this provider referring to other places? What are the top ten pharmacies that they're sending their patients to? What're the overall trends in that geographic area? Is this an outlier kind of payment? So, we try to get some of the questions that we normally get with our risk scores and looking at questionable billing patterns and put that all, package it in one place for our agents to be able to get that information very quickly.

But, we can do this too without our agents, right? I want to make sure that’s clear. This is mostly on them, and I say this too, that if we don’t figure out ways to help them, then we’re failing in our job.

Michael Krigsman: And so, Kelly, how about on your side, you must be going through similar kinds of, or facing similar kinds of issues but with different sets of data?

Kelly Tshibaka: Absolutely! So, I think just to build on some of the stuff that Caryl is saying, we pull our data largely from the agencies that we oversee, because those are… That's where our primary responsibility is. But then, we start with the business question. What am I trying to answer? So, like, to give you an example, we wanted to find out, in the data that we have and the information we have, can we put together any trends or any ideas about how to solve this question? And interestingly, just another shout-out, like Caryl was saying for agents, we can't do our data analytics job without a business understanding of what's going on. So, we have to partner closely with the agents and want the auditors to understand how data are collected, what it's used for, how they understand it. Because, once we understand a little bit more than their business and what they're trying to do, then we can work with the data. Our teams can work with the data in order to figure out how to best use the data and look through it in order to answer those business questions so we get business understanding from our auditors and our agents and oftentimes, even from the people in the agencies that we oversee.

And then, we use that to answer business questions for them and, just like Caryl said, in a lot of ways, I see us like a volleyball team. We’re the setters. We’re positioned, too, on the team. We’re not the one calling the shots and we’re not the one spiking it over that. We’re just setting it up for everybody. Making sure everybody has got what they need in order to make the play that they need to play. That’s what we do and we do it really well. And, it’s exciting to get to use data to do it.

So, to answer your question specifically, Michael, we pull it from the agency, but we also pull it from other agencies if we need it. We’ll pull it from private companies if we need it, and if we can get it. You’ll be surprised what’s even just publicly available on the internet that we can pull and if we can get it into a usable format, then it makes it a lot easier for us to work with it and come up with insights that people can take action on.

Michael Krigsman: We have an interesting question from Arsalan Khan on Twitter, who asks, “Do the OIGs, the people working in the Office of the Inspector General share datasets across agencies and who is responsible for gathering, managing, and distributing these datasets?” Who wants to take that?

Kelly Tshibaka: I’ll take that!

Caryl Brzymialkiewicz: That’s fine!

Kelly Tshibaka: Sure. So, the answer is, “Yes, it depends on the lawyers.” Does that help? Yes, we do share datasets when we can, and usually, it’s the lawyers who are the ones deciding what we can share and how much, because the government has different rules about how it collects data, what it’s used for, and a lot of those roles are governed by the Privacy Act. And, even if we all say we want more data so we can do better oversight of government agencies, we also all know that when we start pulling on that thread too far, we’re starting to really implicate our own privacy rights. And so, there’s this tension in the government about how much information we have, what we can use it for, how long we can use it, where we keep it, what the retention rates are. And, there’s a lot of different agencies, a lot of different lawyers, a lot of different statutes that apply to potential datasets.

So, between the OIGs, yes, we do share datasets when we can, and it’s usually the lawyers who make the decision about what we can share and when. And interestingly, one of the policy challenges of data is that the policy on data changes. It’s sort of an emerging field, and policy usually lags behind emerging trends, it doesn't usually get in front of it. We still have some statutes on record that talk about how we use telephone surveillance, and it talks about telephones and how they were wired in the 1970s. So, most of us don't use phones plugged into our kitchen walls anymore. But, that's how the statutes are written and until things get updated, we're limited in what the policy can do for us.

Well similarly, what we found in the data world is that a lot of times, the policy has to catch up to where the data is. And so, we have to use careful principles of how things worked in the past in order to kind of do our best guess of how to protect privacy and use data now. But then, what we find is we'll build models and we have to revise them when policy changes and [they] let us do more with data, or sometimes, let us do less.

Caryl Brzymialkiewicz: Can I add to that as well? The two things that I would add to what Kelly said is that one, the ID Empowerment Act from 2016 actually gave OIG's special exemptions from some of the Computer Matching Act. So, there are a whole lot of rules; rules and regulations that we have to follow. So, Kelly's absolutely right that we're concerned about privacy; we're concerned about security. But, we also know that, you know, our concern is that if fraudsters are taking advantage of one program, they might be taking advantage of others. And so, how can the OIG community […] with each other and coordinate appropriately?

So, even within that OIG community, there’s Counseled Inspector Generals… CIGIE; what does the “I” stand for? You know…

Kelly Tshibaka: Integrity and efficiency.

Caryl Brzymialkiewicz: Thank you, I was blanking on the "I" for just a second. CIGIE is looking into and giving guidance to all OIGs as to how we can best share that information. So, there are moves within the OIG to share information amongst ourselves. The other thing that I always think about, too, in our organization… A lot of people want to share as much as we can, and I understand that, but I always balance that with we also have our data use agreements with the agencies, right? A lot of our data does come from the agency itself. And so, we have to be mindful that we aren't the generators of the data, and we have to respect how those agreements have been and make sure that we're doing it appropriately.

Now, Kelly talked about the lawyers, but I see it as a very positive way that we’re protecting the information that we’ve been entrusted to look at. Or, we’re still pursuing wrongdoers and holding them accountable.

Michael Krigsman: How do you strike that balance? Because you must be… You know, I can imagine being torn apart in one sense, because you have these dual obligations to protect privacy and at the same time, to catch the bad guys and fulfill your organizational missions. So, I can imagine that this is a very tough balance. So, how do you strike that balance?

Caryl Brzymialkiewicz: I think it’s a constant tension between, I have a specific entity of interest, and so, usually if you think about it in a subpoena case, or I want to go follow something specifically, even agency folks tend to be more willing to give you information. If you already have somebody, a point of interest, right? It’s harder to do this in the Big Data world. That’s where it’s kind of fuzzier boundaries at the moment. And so, it’s an evolving landscape, as Kelly mentioned, right? If I have all of the claims data, then who should have access to all the claims data for trends and outliers? Who should be able to see that? Is it law enforcement sensitive? If we’re combining it with other datasets, what does that mean?

And then, for us, it’s also extremely important to remember that our agents aren’t prosecuting their cases. We partner with the department of justice, very much so. They are the ones that are actually taking these cases and following it to an outcome. And so, a lot of it how much information do we make available to them up front, if we can? Can we make our tools available to them so that they have the same picture as us? But, it’s a balance and it’s a lot of conversation.

I found when I first came to the OIG, I was invited to a meeting with our agents and we were trying to talk about sharing information and everybody fell back to, "It's the technology. We can't do it. It's a tool. It's a tool issue. We have to figure it out." And I said, "Well why don't we just use… We have a secure channel. Why don't we use the secure channel and send a hundred leads? If you think that's really information-sharing…" I don't think it's a technology problem. And sure enough, it ended up being more culture and process. And that was the right conversation to drive, but I think everyone wanted to start off with wanting to talk about it being a technology barrier, when in fact, it wasn't a technology barrier at all. And those are the right conversations we need to have.

Michael Krigsman: And, Kelly, you know, again, in this whole privacy… How do you draw this line? I think this whole issue is such an important one.

Kelly Tshibaka: Yeah. You’re absolutely right. It’s really hard to talk about it in general terms, because it really comes up on a case-by case basis. But, I’d be happy to share with you my philosophy. My philosophy on it is that we hold, just like you’re saying, we hold both of these values as equally important. And so, then, when it comes time to start talking with the people who, like, Caryl said, own the data or we start talking with the people who have the data may not own it, but have it, we start evaluating those two considerations. So, on the one hand, what are the roles? What does it actually say the limitations are? Caryl’s right. A lot of it is read in like the times I’ve heard “Privacy Act” thrown around. “Privacy, privacy;” well, if you open the Privacy Act, it doesn’t say anything about anything you’re saying, but there aren’t a lot of people courageous enough to read it.

So, you kind of have to knock down the fake obstacles and then look at the real obstacles. Something I found, Michael, is that it seems like data… You know how they used to say, “Music and math are universal languages across cultures and languages,” I think, in a similar way, data is becoming that way. It’s becoming like a universal language across government agencies where we can all get behind the same data and say, “That’s what it says,” or “That is the business problem. I have something I can add to that. Can we get behind the mission to find a solution?” And I’m not in any way saying the solution is just more data, more data; I’m just saying that there’s probably more solutions than what we have on the table and if we knock down the fake obstacles and look at the real obstacles, usually, we can solve them. But, it takes everybody kind of having a meeting of the minds to say our common intent is to, for example, stop fraud in our agency. We’re not here to point the finger at anybody. The data says what it says. What are we going to do about it? Or, the data says “this” and we need a little bit more to understand it so we can help you.

Usually, when people have… They know that, like Caryl is saying, that we’re not playing “Gotcha!” We’re really here to try to use data to find where the real criminals are. The enemy isn’t the OIG. The enemy isn’t agency management. The enemy is the person who is taking advantage of our agencies and defrauding them, whether those are external vendors, or they’re even people who work for us. And, we can all unite about that. And, what can we do to find solutions to the problems that are getting in our way of solving that? Sometimes, the law or the culture, or the agreements, they’re just not going to let us move forward. But, it seems more and more, and I agree with Caryl. I applaud our attorneys. We have really creative attorneys who work here. They’re not going to talk about, Yes Attorneys or No Attorneys, or Greenlight-Redlight Attorneys, it’s just too easy to find an attorney who says “no.” But, it’s a lot harder to find out what their real obstacles are. Sometimes, those can’t be resolved. But a lot of times, they can. And then, let’s work together to fix them.

Michael Krigsman: Caryl, you’re nodding your head.

Caryl Brzymialkiewicz: Oh, absolutely! I think one of the things I think about with our analytic team is we obviously… It’s kind of the first mention of “Yes we can” and “How can we?” Right? So, we have a very customer service-focused team, and it’s always just a matter of, “What really are the real barriers? Not just the perceived barriers? And how do we find the right people to pull to the table and have a conversation to work through it?” That’s why I was nodding my head, because we have some fantastic folks in our counsel’s office that help us navigate some of these issues.

A lot of what we talk about to users is the difference between data access, right? IT’s based on the ID Act and the ID Empowerment Act. We have access to data of the agency. That doesn’t necessarily mean system access, which can be a challenge if you’re fighting fraud and you need something very quickly and you want real-time information. One of our successes has been that we do have system access into the Center for Medicaid and Medicare's integrated data repository, which has been huge.

So, an example of that: There’s actually the DOJ press release that you can see. Our team partnered with the FBI to uncover a billion dollar Medicare fraud scheme in Florida. And it was really because we had access to the system to be able to pull a thread very quickly looking at the claims data that we were able to uncover some of the schemes, some of the pattern of the fraud. And so, you know, we talk about the “How can we?” Once we have the system access and are able to look at the data right here, “Here’s what it’s enabling.” And so, I was also taking it back to system owners within CMS and saying, “Thank you so much for working with us. I know that wasn’t easy to let the OIG have access to your system, but here’s now what we’re able to do with it.”

Here's what we've been able to find. Here's what it's empowered. And, I think that's been very helpful both from their perspective, I hope. I think it has been. But also, just to share it with the leadership here, right, within HHS that here's what happens when you do share data. I think some folks are afraid to share data because they're still worried about that it might get misinterpreted. You know, I'm not sure a change is frequent. I want to make sure that you have absolutely the right thing. But, part of it, it's also we try to take a very transparent approach with the folks we're working with to get the data, to run the algorithm, to see the results and then circle back with them to say, "Here's what we found. Is this right to you?" So, if something is misleading in the data for some reason, right? Looked at the wrong variable, something got misinterpreted, and there can be a conversation about that.

So, I think data’s not the only story or the end of the story. It’s usually the very beginning. It’s the volleyball now. Of course, I’m too short so I can’t play volleyball very well [Laughter], but I like that analogy of thinking of us as the setters in volleyball.

Michael Krigsman: And, you’re sitting at the intersection of technology and the business; how the business functions and the law and so, I have to imagine that that makes everything that much more complicated.

Kelly Tshibaka: Well, “complicated” is one word. “Adventurous,” and “exciting” is another. [Laughter]

Michael Krigsman: So, you’re dealing with the data science and you started talking earlier about your teams. And so, from a technology standpoint, who do you employ? You mentioned; Caryl mentioned earlier “Unicorn programmers are needed to find those data needles in the haystack.” I’m assuming you must have data scientists, so who supports you from a technology standpoint?

Caryl Brzymialkiewicz: I’m happy to go at that first. I had a great partner in the CIO Chris Chilbert, who has been absolutely fabulous. I think, when I got here, we were looking at our own infrastructure and figuring out if it was the most effective and efficient within our own organization. So, you know, my team had done a lot. Several years ago under sequestration, it was always the push to do more with less; do more with less; do more with less; and they are just so creative and figured out how to program in new languages, to figure out how to leverage these external systems and environments with the power behind them. And now, it’s really an opportunity again to think, “Okay. What more can we do?” We have a new CIO here. He’s been here about a year and a half now, and thinking about we want to get our infrastructure more resilient. We want to make sure our network’s upgraded. We’re looking into cloud solutions. We’re looking more into open source. We’re looking into mobile capabilities and really figuring out how do we enable our agents?

So, you know, for an example, our agents used to have Blackberries, right? Like, there’s nothing more telling that you’re a Fed that you have a Blackberry. [Laughter] So, part of it was figuring, I guess, […] first we’d pilot it with our agents, but now, that got us thinking when we first created our tools, right? One of them was looking at the payments by geographic area to understand what was happening in their federal judicial district or by county, or by state. Whatever level they wanted, but when we created it, it was meant to be under desktop. It was meant to be on your monitor. Well, after out in the field, where agents are really at their desks very often, they're out talking to people. They have questions. So now, we're thinking about how do we really make sure that our technologies are mobile-enabled to give them exactly what they need and when they need it?

So, it’s exciting, right? It’s not challenging. Like Kelly said, it’s exciting to have that conversation, to pull our digital services director, we have a new person here, too, Evan Lee, who’s been here about maybe a year now? Can’t remember exactly when, too, but I talk about it’s a lot of us are newer to the organization and we’re the change agents. We’re trying to enable people to do more with what we have and we’re being thoughtful and creative about other solutions we can bring to the table. And so, it is a fun intersection of technology and data analytics. And the folks that I have on my team are more the data scientists actually producing the algorithms and coding to figure out how do we take all these complicated healthcare codes and look for the outliers and look for the comparisons. Though, we need our partners very much in our team; OIT, our permission technology group and our digital services to figure out how to evolve the system.

Michael Krigsman: Kelly, what about this notion of being the change agent? Maybe, could you elaborate on that?

Kelly Tshibaka: Absolutely! I should think that ties in nicely with what Caryl was just talking about. Who are we looking for here? We can talk about degrees and backgrounds. You know business scientists, business analysts, computer scientists, data scientists, but the fact is, what I think I found in common with all of our top performers is I think of them as "Imagineers." They're highly creative people who are like engineers with data or with computers. They know how to use technology to answer these questions and to kind of… They use it to explore and to do basically whole investigations, or full audits […] from their computer. They can just dig into all of this information and they do it with a lot of joy and enthusiasm.

But interestingly, when we talk about the change agent piece, if we just think about data scientists or our data analyst division, a Chief Data Officer office of just being a bunch of data geeks, you’ve totally missed it. What I think has worked really well for us, our IT thought outside the box and said, “We need to pull in,” again, the key here is understanding what this is being used for and how to use it. We need to pull in these creative ambitious people from around the agency and get them all in this one think-tank; this creative tank known as CDO. It’s like, the Switzerland in the camps of audit and investigations. You’re not going to see CDO and AIG […].

So, anybody who comes here kind of has to leave their audit or investigator title at the door. They don’t actually, but they come with that background into CDO where we all get around a table or get in front of a board, or get in front of a computer screen and figure out solutions to problems. So, our team’s agent; the way we worked is once we saw that we could actually do stuff with the data; the data was valuable and we could turn it into results for the organization. Our goal is to make the OIG better to improve its return on investment in audits and investigations. We’re doing investigations and audits faster. They’re higher quality they have better turnaround because of the data.

Like Caryl was saying, one of our goals in this era of “Do more with less,” is to use analytics to do that. I really think analytics is the solution of how do we do more with less? How do we make sure auditors are directed to the highest value audits? Make sure that we […] 500 leads to investigations last year, only one of them was unfounded. All of the others had merit. And that’s a huge win, just saving time and effort for our agents that they know that they’re looking at stuff that’s valuable. So, when we get all of those people; auditors, agents, people with inspection backgrounds, a lawyer, accountants, fraud examiners, MBAs, all of these people are in our group. Get ‘em around the table, they approach the problem from different perspectives and that’s how change happens.

Not only do we come up with more creative solutions that if you think of a ship, we’re able to inform the executive how to direct the ship. Like, let’s go in this direction. But, we also can weaken our support to organizations with the data analytics. We can grow in them back and guide from the front. It's a really great place to be, but I think, really, the key is that we do it with a lot of creativity. We do it with some knowledge expertise, whether that's in data, or in business, or actually in the work that we do like investigations and audit. That's how we've been able to do change here at OIG.

Michael Krigsman: It’s really interesting what you said, that data and analytics are the keys to doing more with less. And, you touched on some of the metrics and KPIs that guide your work. And, Caryl, what about on your side? Can you talk about how do you measure what you're doing and the outcomes that you're on the right track?

Caryl Brzymialkiewicz: So, part of our effort, actually, from my office is also just to establish parity outcomes and keep performance that caters to the whole organization and then to figure out better ways of tracking it internally to see if we’re meeting our metrics. Very specifically, for analytics, one of the things I was really pushing on was to understand if we’re developing all these tools, how many people are using them and what kinds of things are they pulling to inform the work? And so, we had to create tracking mechanisms to understand that. We reported out on that routinely. We have internal meetings within our division that we talk about that, and we actually put it out to the organization to say, “Here is what we are delivering on your behalf. Here’s what we’re trying to do.” And, it’s also just to make sure that people are aware. The part that I was also reflecting on; Kelly and I talked about this; a lot of it also has to do with communication and education in the organization.

If you say data at the fingertips of the OIG, that can mean multiple things to multiple people, right? So, which data are you talking about? Is it our internal data? Is it the external mission data? Are we talking about data for executives in management? Are we talking about data for our front line auditors, evaluators, or investigators and attorneys? What are you talking about? It’s a huge conversation. So, what I found as some of the metrics we’ve put in place have actually made… Well, that conversation mostly to help with the education and communication about what we’re trying to do. Overall, my goal is to make sure, as an organization, our specific mission is to provide more and better access to data analytics.

And, my specific metric is trying to reduce the time it takes for all of our folks to access quality data. That is loaded, right? There were lots of conversations about how are we going to measure it? And, I still don't think we've quite gotten there on the how do we measure our quality data for some of these pieces? But, we're working on it. And then, a point that we try to derive what we need to create these metrics so we make sure tracking to the right thing, and we just keep talking about that priority, and our mission, and how we're trying to accelerate our analytics are trying to use data to drive more firm decisions and just enhancing our use of data.

There’s data all over the place. It’s how do we pull it together and make sure we talk about this too? How do we make sure that OIG knows what OIG knows, right? So, we don’t have ten copies of the same spreadsheet that it’s appropriately shared, appropriately stored, appropriately controlled. And so, those are kind of some of the things that we’re tracking.

Michael Krigsman: It’s interesting. It sounds like Caryl, your focus, you’re describing your metrics and KPIs as relating to the data and how the data is then used. And Kelly, you were describing yours as speed of investigations. And things like that. So, is there… Are those different? Are you saying different things? Are you saying the same things in different ways? Does it reflect different focus for each of you in your particular agency?

Kelly Tshibaka: Yeah. I think one of the things Caryl and I have talked about, we’ve worked together in sharing what our respective performance measures would be because we both have a role to play in developing those for our agencies. And, I think that you’re right, Michael, that in a way, we are looking at similar things in a way we’re emphasizing slightly different things. But, I think, ultimately, we’re aiming towards the same goal. We have to take our agency, the place where it is now, and help guide it towards being the best possible OIG it can be. So, one of the things that we decided to look at for data analytics, we know we’re going to be successful and helping our agents if we’re reducing the time it takes for them to make successful cases.

So, it used to take around three hundred, or 530 to do a case, on average. And now, we have it down to under 390 when they’re using data analytics. Or, we used to have a return of about 600,000 dollars on average in financial impact on a case. And now, it’s over 900,000 because of the data analytics. And so, we’re trying to measure exactly how the data analytics is helping. We also look at the cases and the audits that are performed as a result of the tools that we’ve developed.

And so, one of our models was an audit model because you wanted to talk about fraud. So, we developed an audit model looking at contract fraud; pulling data from different areas. You know, looking at things that we would consider like contract pricing of the CEO’s role and then contract; just looking at different elements of contract. And, we pulled it together into a model for auditors to be able to really quickly rack and stack the thirteen billion dollars in contracts the postal service has every year so they can identify which contractors or which contracts had the highest probability of fraud. And in one of those years, there was a 500 million dollar return in our audit findings just from that one model. So, those are the kind of things that we’re looking at over here at the OIG.

Michael Krigsman: So, it’s enabling you to not just conduct these investigations more efficiently, but it sounds like the data enables you to do new kinds of investigations to kind of rethink the type of investigations you do, because of the availability of that data and the type of analytics that you’re performing on that data. Is that correct?

Caryl Brzymialkiewicz: I think what we talk about is, in particular, Medicare fraud if you think about Florida, Miami, there’s so much fraud… This is not about generating leads for our agents, it’s really about how can we help an optimization of the work? So, as Kelly was just talking about with the optimizing audits, where do we want to point our resources? So, for our organization, if you think about a trillion dollar portfolio that you’re supposed to provide an oversight for, we have about 1,600 people in the whole Office of Inspector General, and about 70-plus offices throughout the country. That’s really not a lot of people but that size of portfolio… We’ve done a bubble chart of our budget compared to the whole budget of HHS. It’s not big.

So, what it really needs is we're trying to focus the efforts as best as we can and help inform those conversations, right? Data are not going to be the only input into deciding which audit to do, which evaluation to do, or which case to go after. But, it can be helpful. And, that's really the way we looked at our predictive analytics with our risk models. Does it help find maybe a risk of… There's a high-risk provider that's not yet under investigation, or it's validating that when we do run our models, we compare it against our case management system and find that there actually are a lot that is already under investigation. And then, in that case, we look for other linkages and what's the link in analysis? What’s a network analysis? What else can we add to those cases? That may be an additional insight that we hadn’t seen before, and that’s why I think we’re really trying to add value in that case.

Michael Krigsman: It’s really fascinating. We’re almost out of time and so, I’ll ask each of you maybe Kelly, want to go first for your final thoughts on the use of data in these kinds of investigations that you’ve been describing?

Kelly Tshibaka: I think that the data analytics frontier just represents so many incredible possibilities for doing oversight work of federal agencies in new; and I agree with you, Caryl; optimized ways. We want to find, as effective and efficient ways as possible, to do our job so that our auditors and our agents get to work the best cases and are working on the highest risk program areas for our respective agencies. And that’s just a win for us. I mean, I think that with what Caryl and I are doing, we’re really exploring that frontier and turning it into a reality for our respective OIGs.

Michael Krigsman: And Caryl, it looks like you’re going to get the last word. Maybe, share with us your thoughts on being a change agent in the government since clearly, that’s a very, very important function that both of you play.

Caryl Brzymialkiewicz: Change management is not an afterthought. It should be the first thought and everything that we do, no matter where we are in the federal government… You know, we talked a lot about the “How can I?”, but the questions I love the most are “What if?” and “Why not?” And if you start from that, you can get some momentum going behind you about how can we make this better? I tell my team all the time that we are the change agents, the innovators, the enablers, and the accelerators. It’s not that there’s something broken in our organization that we need to fix, it’s we need to take what’s working and make it better. And so, any way that we can enable that and help our folks do their jobs even more, that’s when we really know we’re connecting to the mission, right? We want to hold wrongdoers accountable. We want to make sure that we’re protecting the programs.

And so, being change agents; you know, sometimes I call us the gentle agitators; because, that’s really our role is to ask that question of “Why not?” and “What if?” And so, really, I’m inspired to […] on my team. I have a fantastic team and it’s really about the organization and the partnerships we’ve had here with the business. I’m very appreciative of all of our investigators and our auditors and our evaluators that work with us to figure this out or without them, we would not be successful.

Michael Krigsman: Okay! Wow! What an interesting conversation and I love that the gentle instigators… Well, you have been watching Episode #253 of CxOTalk and we have been speaking with Kelly Tshibaka, who is the Chief Data Officer of the U.S. Postal Service with the Office of the Inspector General. And, we’ve also been talking with Caryl Brzymialkiewicz, who is the Assistant Inspector General and Chief Data Officer at the U.S. Department of Health and Human Services, also with the Office of the Inspector General. Thanks so much for watching CxOTalk! Be sure to “like” us on Facebook and also, subscribe on YouTube, and we will be back with more next week. Go to and check it out. Thanks so much, everybody. Have a great day. Bye-bye!

Published Date: Sep 08, 2017

Author: Michael Krigsman

Episode ID: 468