Kevin Neary’s 2024 Mission: Crushing AI Bias

As artificial intelligence continues to transform our world, the conversation around its ethical implications is increasingly urgent. This week on CDO Insights, we had the privilege of delving into this vital topic with Kevin Neary, a renowned expert in Responsible AI. Kevin’s unique blend of insight and inspiration challenges organizations to think critically about the ethical dimensions of their AI initiatives, focusing on the essential principles of ethics, transparency, trust, safety, and sustainability.

Kevin Neary stands at the forefront of the Responsible AI movement as the CEO of Orcawise. With an international reputation as a keynote speaker, he captivates audiences with his engaging and informative presentations that bridge the gap between technical complexities and business realities. As a steering board member at CeADAR, Ireland’s national center for AI, and an advisory board member at Georgia State University’s AI Lab, Kevin is deeply committed to promoting ethical AI practices across industries and educational institutions.

Key Highlights

During our insightful discussion, Kevin illuminated several key points, including:

• Ethics as a Cornerstone: He articulated the necessity of embedding ethical considerations in AI development to foster trust among users and stakeholders.
• The Power of Transparency: Kevin stressed the importance of transparency in AI processes, advocating for accountability to enhance public confidence.
• Proactive Risk Management: He provided strategies for identifying potential risks associated with AI technologies, ensuring their safe deployment in various applications.
• Commitment to Sustainability: Kevin championed sustainability as a critical factor in AI practices, urging organizations to align their technological advancements with environmental and social responsibilities.
• Inspiring Action and Change: Throughout our conversation, Kevin inspired listeners to take meaningful actions toward embedding responsible AI principles in their organizational cultures, paving the way for a more ethical future in technology.

Kashyap: Hi Kevin, thank you for making the time to join us. When we first spoke, you shared your passion for the subject of responsibility in AI, which I find incredibly important. As someone who started as a data scientist and has spent the last seven or eight years in the AI and data science space, I’ve witnessed the dialogue around responsible AI evolve significantly. It’s great to see you choosing to address this topic, especially as we delve into the critical issue of bias in AI systems. Before we dive deeper into that discussion, could you introduce yourself to our audience? Tell us about your background, the founding of Orcawise, and the specific work your team is doing to promote responsible AI practices in organizations today.

Kevin: Yes, absolutely. I’m the CEO and co-founder at Orcawise. As you pointed out, we are an advisory services firm. We work with clients, mid-sector and enterprises, helping them ensure that their AI systems are responsible. We are a university spin-out from University College Dublin. In 2016, we started off as being AI for marketing, and then we progressed into responsible AI because we identified that there was a big market coming down the line for responsible AI systems. Right now, we are a team of over 60 data scientists and legal and compliance professionals all working together to help our clients have responsible AI. We operate across the United States and in Europe, and we’re very much aligned around the EU AI Act and help companies be compliant with the EU AI Act. That also involves bringing in legislation from other countries like the US. Obviously, a lot of guidelines exist in the US even though there’s not a lot of legislation. The legislation is in Europe, and we find a lot of our work is around helping companies to navigate both jurisdictions and try and have systems that are responsible.

Kashyap: To dive deeper into your consulting services for responsible AI, could you explain what that entails? I’ve noticed that some companies establish standards or provide a stamp of approval for algorithms, while others focus on legal consulting. If, for example, a banking client approaches you about developing a customer-facing chatbot, what does your process look like? Do you provide consultancy, or is there an approval process with standardized assessments? Additionally, do you work at the company level, evaluating responsibility for AI applications, or do you also assess the overall AI systems within organizations?

Kevin: So initially, we started out as an AI consultancy developing AI applications and generative AI applications and then we shifted to responsibility. And so our focus now is around responsible AI. So somebody else might be delivering the generative AI application and we say, okay, you need to ensure that you are responsible, ethical, and transparent with an application that you’re building or maybe somebody else is filling it for you. So, we work at that layer whereby we have the companies think about responsible AI first so that they lay a solid foundation around responsible AI.

The driving force behind that is legislation, and primarily the EU AI Act, which is applicable to U.S. firms and European firms. And really what we’re looking at here is ensuring that high-risk systems are compliant with the EU AI Act. So if a bank, for example, is working on a chatbot and the chatbot is interfacing perhaps with sensitive data, private data, and if a chatbot was endeavoring to interface with human emotions, for example, that’s a high-risk system. Therefore, the application needs to be built with responsible AI in mind.

So in order to do that, we need to work with lawyers and work with data scientists. I’m not a data scientist. I’m a business person, so I’m very much at the intersection of data science and business, and the business in this case, from our point of view, is legal and compliance professionals. And of course, in the example you’re providing, it’s the banking operation as well. So, we bring in all those three entities together and set up a solid foundation to ensure that whatever is developed is responsible and is compliant with the various legislations that might apply to that.

Kashyap: So basically it’s the role of lawyers and data science professionals both is very important to you and the kind of consultancy that you are offering.

Kevin: Absolutely, the lawyers, compliance people, and data science people are all critical to the process. So we need lawyers to interpret the legislation and the EU AI Act. If there are guidelines from the United States in different areas, we need the lawyers to interpret that and then work with the data scientists to understand the intersection of both of those. Then we work with the clients to understand exactly what their goals and objectives are—for example, for that chatbot—and lay that foundation down. So it’s about bringing all of those teams together and having one intersection that is focused from our point of view on Responsible AI.

Kashyap: Thank you for laying the groundwork for what you’re doing. I’d like to shift our focus to some practical examples and discuss the topic of bias within the broader context of responsibility, which includes elements such as fairness, accountability, transparency, and explainability. However, today, I want to specifically address bias, especially in high-risk applications.

With 2024 being a critical year as we’ve been immersed in the Generative AI buzz for some time now and have seen significant evolution, can you help me understand how the roles of your data scientists and lawyers have evolved from 2020 to 2024 in helping companies address bias? Considering that many AI models were static back then and now have transitioned to dynamic training, where models continuously adapt based on new inputs, how do you address bias in this context? What strategies do you employ to consult with your clients to ensure they tackle this issue from the very beginning?

Kevin: It’s a very interesting topic, and you are right; things have evolved incredibly over the last four years. It’s all about creating awareness amongst the clients around the whole area of bias, and that’s an area I’ve become fascinated with. If we think about bias, first of all, from a human point of view, we have what we call human bias. We all have biases in us. It turns out there are around 188 different cognitive biases that can sneak into our thoughts every day as humans. So, it’s no surprise that AI systems can pick up a bad habit or two. If we think of AI as a living thing, we know it learns and changes over time, which means new biases can emerge after a system is deployed. Now, while human bias is part of our nature and is important because our biases help keep us safe and happy—there are good things about it—AI bias is a different story altogether. AI bias can scale across millions of decisions, affecting fairness, equality, and trust. As you know, these biases can lead to unintended consequences on a large scale, impacting organizations in financial services and healthcare significantly. It’s an important question that you ask. In terms of engaging with clients, we bring this example of human bias and ask them to reflect and consider how easy it is for bias to seep into AI applications. That’s how we get the conversation started.

Kashyap: How has the journey from static to dynamic models impacted your approach? Given that the roles of your lawyers and data scientists must have evolved significantly, can you elaborate on this evolution? Earlier models were quite static, trained on specific datasets, and biases were addressed both technologically and legally over time. With the advent of dynamic systems, including reinforcement learning and rapidly evolving models, how do you assist your clients in addressing biases? Could you discuss both the technical and legal aspects of this process?

Kevin: Detecting bias at the development stage is one thing, but then we got to move on to continuously monitor and address the biases as they seep into the systems, as we identify them and find them. With our clients, we put forward a number of steps that they should consider, and that we would support them with.

First, I’d mention continuous monitoring and feedback loops as crucial techniques. Real-time data analysis is also important when it comes to bias. Most importantly, having a very strong and advanced data strategy is critical.

If we look at the first one, continuous monitoring, think of it this way: just imagine that you’re checking your GPS regularly throughout the day to make sure you’re on the right path. In the same way, we need to constantly check AI systems to detect bias early and correct them before they do any harm.

Another example that I was working on recently in the hiring industry for a financial services company is that the AI systems started favoring certain resumes over others. By implementing a simple automated continuous monitoring system, we enabled the organization to catch that bias early and then start to fix it. I think feedback loops are really important. I talk a lot about the human in the loop and automated feedback loops, and you can think about this in a GPS way as well. Think about your GPS recalculating when you make a wrong turn—so feedback from real-world users helps AI adjust and fix biases, especially those that aren’t obvious in training data but show up during actual use.

And that evolution you talk about from 2020 to 2024, the old training data has picked up a lot of biases, which needs to be highlighted as we move forward. I mentioned real-time data analysis, which is similar to checking the weather forecast throughout the day, not just in the morning. I’m in Ireland right now, where it rains all the time, so I’m checking all the time. But that always reminds me that AI systems need to be checked that way as well, constantly.

Spotting bias patterns is how we think about this. A lot of organizations think about spotting isolated pieces of bias or simple anomalies, but really, we need to be looking for patterns, especially in high-stakes industries like financial services and healthcare, which we look at very closely.

And then, the data strategy that we mentioned is critically important here. This is like having a balanced diet with regular checkups. Training AI on diverse data reduces bias, and regular audits ensure the system stays aligned with ethical standards. So, this style of dynamic bias detection, continuous monitoring, and using different steps and techniques in order to continuously monitor is really important. I often think of it as doing everything possible to make sure your GPS does not send you into a lake. We have to keep AI on the right path.

Kashyap: As we shift towards more dynamic models, generative AI has become a critical topic, despite some fatigue around the discussion. Addressing biases in generative AI is essential. You’ve mentioned the transition from static to dynamic models, focusing on increased monitoring and technological advancements.

Given that generative AI relies heavily on vast, uncurated datasets sourced from the internet, what unique approaches is your team implementing to detect and mitigate bias in generative applications?

Kevin: Yeah, it is an important distinction that you make and it’s important to look at generative separately from AI and in a broader sense. Because I don’t think a lot of organizations are fully aware of how much bias can creep into generative AI. You mentioned there and the Internet. And if you think about generative AI and how AI works, it mimics the quirks of the data it’s trained on.

Generative AI can be like a friend who’s desperate to impress you; for example, if you like pineapple on pizza or think cats are better than dogs, so does your AI. The problem with generative AI doesn’t stop there—it scales these biases and sometimes broadcasts them from virtual rooftops. And this becomes especially concerning when generative AI models are working with text and images that are trained on massive uncurated datasets from the Internet, to your point. It’s like they’re on an endless binge of online trends, absorbing both the best and the worst and everything that’s out there.

From my discussions with business leaders, I found that organizations are starting to tackle this issue in creative ways. If you want to look at those creative ways that I’ve come across, curating quality training data is very important here. Think of curating training data like putting your AI on a healthy diet. If you feed it junk food, you can’t expect it to be a health guru. Carefully selecting high-quality diverse data teaches AI to see the world in a more balanced way, therefore reducing bias.

For instance, I got first-hand experience of this problem when I was prompting one of the large language models for details about AI legislation in the US and in Europe. The responses I was getting from the model were very general; they were useful but very general and not nuanced enough for something as important as ensuring compliance with the EU AI Act. That led me to look at this problem, and we thought about building a more focused model with curated data relevant to the data that we had to invest in.

So, what we did was we built a system of data curation with lawyers and we’re creating 25,000 questions and answer pairs. This is our curated data created by humans, and we’re using that to train a custom model on top of one of the frontier models, and we’re getting great results from that. It’s time consuming and a big investment, but what’s coming from that is amazing.

Kashyap: You mentioned the importance of human intervention and monitoring in algorithmic auditing. How effective are current auditing frameworks in addressing these issues, and what improvements are needed?

When building systems to track biases, could you provide a specific example of what exactly is being monitored? Bias can be subjective; for instance, discrepancies in hiring rates between males and females may not always indicate bias.

In this dynamic AI landscape, how do you conduct algorithmic audits, and what frameworks do you use? An example would greatly help our audience understand this better.

Kevin: Absolutely. So the one thing just before moving on to the algorithmic frameworks, which you highlight as being really important—and I agree—is to have diverse teams in place inside your organization. A lot of these biases originate with humans and evolve over time through our practices. Therefore, I would always encourage organizations to look at their teams very carefully. Most companies are very interested in diversity and inclusivity, and that’s a great opportunity to improve the AI team from a bias detection point of view as well as from a development point of view.

You are right; there’s a lot of talk around algorithmic auditing and building frameworks, and the importance of this cannot be overstated. These systems ensure that AI remains fair, transparent, accurate, and accountable, like you said. Audits are becoming more common, and one thing driving these audits is the amount of legislation and guidelines that are out there. You can correct and set out a whole range of challenges around correcting bias with these audits and the auditing frameworks we have now, which often focus only on technical aspects like quality, model performance, and the accuracy of algorithms. Of course, all of those are crucial, but they don’t always catch the more nuanced biases related to social and ethical concerns.

Another issue we see a lot, particularly in large organizations, is the effectiveness of the audits that are being run. The effectiveness of these audits heavily depends on the quality of the auditors and the tools used. If you don’t have the right experts or bias detection tools in place, these audits might not fully address the deeper societal issues that may be embedded in AI systems. I think we are moving into an era where these social and ethical issues and characteristics are becoming more and more important.

I see many tools in the early stages being developed to address this piece correctly. Generally, we need to raise the bar by incorporating ethical and social impact assessments into these frameworks. These assessments specifically tap into ethical and social issues, allowing us to catch biases that traditional technical audits might miss. I believe it will be vital for organizations in the future to have more robust processes that assess both technical performance and the wider social impact of their AI systems.

I’ve also noticed in the market that there are tools emerging now. Some of these tools have been around for a while in academic settings inside universities and are slowly creeping into the industry. Explainability tools like LIME and SHAP, which are fairly well-known, help auditors understand why an AI model makes certain decisions, providing that much-needed transparency. Certain bias detection algorithms are becoming more advanced, allowing us to identify hidden biases that were previously difficult to uncover.

Of course, as I mentioned, the biggest driver of change is regulatory standards. This is why we decided to go into responsible AI and bias detection. Because standards like the EU AI Act and the proposed legislation in the US, such as the Algorithmic Accountability Act, are coming down the line. These regulations are starting to ask for AI audits as a requirement, which is pushing companies to maintain fairness and transparency in their AI systems. I’ve also noticed that organizations like the IEEE emphasize ethical guidelines, hiring independent audits to ensure AI systems don’t just work but work responsibly. We are moving toward this societal impact, which is becoming more and more popular.

Kashyap: To add to that point, you also mentioned the assessment of social issues. I want to give our audience a deeper understanding of this topic. Can you provide an example of how you have evolved from simply monitoring the basic outputs of a model to assessing social issues? How do you navigate this entire problem statement, considering that social values can vary significantly across different geographies, ethnicities, and religions? A specific example would be really helpful for our audience to grasp this complexity.

Kevin: In terms of social issues, it’s important to look at how people think about and perceive differences. The appointment we talk about a lot is the intersection of the characteristics of individuals, and we often talk about race, culture, or gender when we’re discussing AI bias. But really, we’re looking at the intersection of some of these characteristics, and that’s very important. That goes some way toward identifying what social issues might impact a certain community, for example.

So, when we’re thinking about one jurisdiction, perhaps we’re concerned about gender bias. But really, in another community, we might be more interested in gender and income and other background issues that might need to be taken into account. Working through these is very complex. I currently have a project running in a research institution whereby we’re looking at this issue and trying to figure out a way to move forward with it.

When I started researching it, we did not have an example of a real-life industry situation that we could follow, so we thought we’d move this to research ourselves. We are a research backed organization. We came out of research, so anytime we encounter something, we go for a research approach. I don’t believe that we have a lot of work done on this right now, Kashyap. I think this is something that we got to embrace and move forward with very quickly.

Kashyap: Let’s move on to a slightly different topic. While we were discussing the assessment side of AI models, which typically occurs at the end, I want to shift our focus to the inputs of these AI models, especially in the context of 2024, where synthetic data has become a significant conversation.

This topic is particularly fascinating because it has multiple layers. Synthetic data is not just about input; it is, in itself, an output of previous models, which generate synthetic data to create different instances. I’ve seen various tools developed to ensure that synthetic data reflects a diverse range of distributions across different responses that feed into newer models.

So, my question for you is: how has this generation of synthetic data impacted your conversations around bias? What are some of the benefits of using synthetic data to mitigate bias, and what are its limitations in addressing this problem?

Kevin: Absolutely. It’s a very good point the way you put it; it’s an input and an output at the same time. Synthetic data, by definition, is like creating a virtual twin of real-world data. It looks and acts like the real thing but doesn’t contain any identifiable information. This synthetic data is increasingly being used to mitigate bias in AI models by filling gaps in datasets that lack diversity.

For example, in a real-world dataset that doesn’t represent certain ethnic groups, synthetic data can step in to help ensure that AI systems remain inclusive. In healthcare, synthetic data can be used to ensure that facial recognition systems accurately identify people of all skin tones, thereby reducing racial bias. This approach ensures that patients are treated fairly regardless of their background. Another benefit is in the realm of data privacy; since synthetic data doesn’t include any personal information, it minimizes the risk of privacy breaches, making it safer to use while providing realistic training data. 

However, you do need to have a system in place to avoid building bias on top of bias, which is the biggest risk with synthetic data. Additionally, if the algorithm generating synthetic data is biased, it can even introduce a whole range of biases that weren’t present in the original datasets. It can take on a new life and not just perpetuate simple biases but can actually start generating a new pathway of biases and reinforce previous prejudices.

To address this problem, developers need to ensure that they are using unbiased well designed algorithms for generating synthetic data. Comparing synthetic data to real-world outcomes is essential to identify discrepancies. Regular audits and human oversight are critically important to ensure synthetic data aligns with standards. The more I look at this bias problem, the more I see the importance of human oversight working with these tools.

Of course, we need the frameworks, strategies, and applications to work on, but the human in the loop is crucial when it comes to bias. You mentioned earlier that financial institutions use synthetic data, which is handy for assessing credit. By ensuring this data includes a diverse range of financial profiles, the particular financial organization can run bias tests on it. This thorough review process ensures the AI model doesn’t unfairly score certain groups. The goal is to create fair AI systems, and as you pointed out, Kashyap, the key point around synthetic data is to make sure we don’t introduce new issues.

Kashyap: With regards to intersectional bias, we’ve touched on various aspects of it, but there’s been considerable discussion lately about tools like the residualism tool developed by Joy Buolamwini. Her work in facial recognition technology revealed that certain racial groups were detected less accurately than others. One solution to this issue is the need for a larger dataset that better represents the ecosystem.

In today’s context, where many AI systems, including general-purpose ones like ChatGPT, are being developed and integrated into various online platforms—be it social media or other applications—how do these companies effectively address intersectional bias? What methodologies and considerations should they keep in mind, beyond just data gathering? For instance, how can they ensure good representation through synthetic data and what other steps need to be taken to mitigate intersectional bias?

Kevin: Absolutely, and this goes back to your earlier question around the societal thing and how to get at that. I think intersectional data is going to be one of the topics that will support us in dealing with societal issues as we go forward. Companies typically look at this in a two or three-step process, and I noticed that intersectional data, which is the multifaceted intensities of individuals, is really important to have and that we can use to consider all of the factors.

So when we talk about intersectional data, we’re talking about how we can have an overlap on different criteria like race and social and geography, all of these things coming into play at once. It’s important to have a strategy that basically deals with intersectional data and to work with that, as you mentioned, in a synthetic data context in order to get that right. Another approach I see out there is intersectional critical algorithms, and I’ve been researching this recently. This is a new technique for me, where these algorithms are designed to go beyond surface-level categories and consider the combined effects of different identity factors.

In this case, we’re simulating what might happen if we have all of these intersectional characteristics coming together. I’ve also come across talk of advanced ethical AI practices, for example, fairness-aware machine learning, which are making strides in addressing these biases. We see a trend amongst companies and particularly researchers right now developing fairness auditing tools that can test decision-making processes, ensuring they don’t disproportionately harm marginalized communities.

It brings us back to your point around social issues, and I think the human loop with that and collaboration between technologies and social scientists is crucial here. Engaging with communities that are already affected by intersectional bias can help AI developers learn from these experiences and build more inclusive systems. So I strongly believe that by involving these communities and tapping into these societal nuances, and then working that into our data, we can ensure that AI benefits everybody in the long term.

Kashyap: And now to my final question: Throughout this conversation, one of the themes you’ve maintained strongly is the importance of having a human in the loop. You’ve emphasized that human oversight is critical, even while algorithms track the outputs of AI models to detect bias. However, AI is fundamentally designed to reduce human intervention, aiming to automate tasks intelligently for maximum benefit.

For instance, consider self-driving cars like those from Tesla, which require drivers to keep their hands on the steering wheel, even though the AI handles most driving tasks. In contrast, there are fully autonomous vehicles operating in places like San Francisco without any human intervention.

Given this context, do you believe that the problem of bias can ever be completely solved, or should there always be some level of human intervention?

Kevin: So this whole question around human intervention is one of my favorite approaches to bias right now, and I don’t see beyond that at this moment. The idea of automating everything around bias seems fairly far out in the future for me. Certainly, we can establish thresholds when it comes to automation. We can look at a predictive algorithm, for example, and think about that in predicting patients’ health. If we look at a prediction scenario that can deliver 80% plus positive results several times, we might consider that acceptable. But what if we look at it the other way, and it’s always under 80%? Then we’ve got a threshold there, whereby we need to have human intervention.

So, having these discretionary systems is obviously very important across all industries. I don’t come across many examples where AI can take care of everything, and I certainly don’t come across any examples where bias can be totally detected, mitigated, and planned for without a human in the loop. On this piece around human intervention, it’s about finding the right balance between human oversight and AI-driven decision-making.

I mentioned before in Orcawise, a firm I lead, where we found collaborative development to be key. Having lawyers work closely with data scientists during the training phase of the EU AI Act custom model that we’re developing is crucial. Our teams are responsible for reviewing the training data and outputs to ensure legal and ethical considerations are included. We run regular feedback sessions to allow for real-time identification of biases, and that all keeps the project moving forward slowly.

Some might say this slows down innovation, but in truth, if we run into a roadblock because we’ve automated everything and then have to start all over again, that would be a bigger detriment to our projects. The human in the loop is twofold: it helps manage the bias situation and ensures that projects move forward. Sometimes at a steady pace, which may not be as quick as desired, but at least you’re getting the human in the loop, and you can be certain that the outputs coming down the line are what you want.

Explainable AI is also crucial. Explainable AI tools are user-friendly and relatively easy for humans to work with. IBM has some excellent tools in this area; AIX is a very good tool that helps users understand why models make certain decisions and help users on how to spot and correct biases in certain kinds of data. Some best practices in this area include clear oversight, and we’re back to the human element. We’ve just started an AI ethics committee in our company, which includes business leaders, legal experts, and data scientists who review outputs and consider the ethical implications.

All this helps ensure we balance innovation with compliance. I mentioned decision thresholds, and I think almost everything I look at in this area has decision thresholds front and center. How far can we go with automation? What is the cutoff point? Certainly, in critical systems like health, finance, education, and even government, which is moving towards AI for many decision-making processes, understanding these decision thresholds is going to be extremely important. This will help us grasp the complexity of the cases we’re working on and understand clearly which cases can be automated in a secure way, with no concerns, and which ones are risky and therefore require human involvement.

Kashyap: On that note. Thank you so much, Kevin, for making the time. It was a wonderful conversation. I know we decided to have a short conversation, but I believe that to do justice to the topic, we have to kind of discuss all the elements. I’m sure that everybody will find it interesting. Your insights were really insightful.