Subscribe to our daily and weekly newsletters to stay updated with the latest industry-leading AI coverage and exclusive content.
AI usage in consumer-facing businesses is growing, as is the concern about long-term governance of the technology. This concern has intensified with the Biden administration’s recent executive order mandating new measurement protocols for AI development and usage. Both AI providers and regulators are currently emphasizing explainability as a crucial element of AI governance. Explainability allows people affected by AI systems to understand and challenge these systems’ outcomes, including biases.
While simpler algorithms, such as those used for approving car loans, can be explained more easily, newer and more complex AI technologies present a greater challenge. Despite their complexity, these advanced algorithms offer significant benefits. For instance, OpenAI’s GPT-4, trained with billions of data parameters, can generate human-like conversations that are transforming industries. Similarly, Google Deepmind’s cancer screening models leverage deep learning to accurately detect diseases, potentially saving lives.
These complex models often make it difficult to pinpoint where decisions are made, which raises the question: should we halt these partially explainable technologies if they can provide significant benefits and minimize harm? U.S. lawmakers are beginning to understand that focusing solely on explainability might not be the best approach for AI governance. Instead, an outcome-focused approach might be more suitable for managing this complex technology.
Uncertainty is not new when dealing with novel technologies. In medical science, identifying and mitigating potential harm from new therapies has led to the development of randomized controlled trials. In these trials, participants are divided into treatment and control groups to observe causality by comparing outcomes. This stable testing design has effectively assessed the safety and efficacy of therapies over time.
In contrast, AI systems are continuously learning, introducing new benefits and risks each time algorithms are retrained. Traditional randomized control studies may not be suitable for assessing AI risks, but similar frameworks like A/B testing can measure AI outcomes continuously.
In product development, A/B testing has been widely used to measure the impact of different product features. This method involves randomly assigning users to either the current version of a product or a new version and monitoring the outcomes. This approach, introduced in online continuous experimentation by Ronny Kohavi at Bing, ensures that observed outcome differences are due to the intervention rather than external factors.
Companies like Bing, Uber, and Airbnb utilize this framework to iteratively improve their products and user experiences, building infrastructure to manage thousands of experiments simultaneously. This iterative testing system can measure not only business benefits but also potential harms like disparate impact and discrimination.
Effective AI safety measurement involves continuous evaluation of AI systems’ outputs on different populations. For example, a bank concerned about potential gender bias in a new pricing algorithm can set up an experiment comparing the new algorithm’s outcomes with those of a benchmarked model used previously. By ensuring demographic attributes are evenly distributed between treatment and control groups, the bank can measure any disparate impact and assess the fairness of the AI system.
Controlled rollouts of new AI features can also help mitigate potential harms by gradually exposing the AI to more users or limiting exposure to less risky populations initially. Microsoft, for example, uses “red teaming” where employees test AI systems adversarially to identify significant harms before wider release.
Measuring AI safety ensures accountability, offering a quantitative framework to determine if an AI algorithm is harmful. This process makes the AI provider responsible for the system’s proper functioning and ethical alignment. While explainability remains crucial, continuous measurement techniques from healthcare and tech can help ensure AI systems work as intended and are safe.
Caroline O’Brien is the chief data officer and head of product at Afiniti, an AI company specializing in customer experience. Elazer R. Edelman is a professor at MIT and Harvard Medical School, and a senior attending physician at Brigham and Women’s Hospital in Boston.