Sign up for our daily and weekly newsletters for the latest updates and exclusive content on leading AI coverage.
Last month, I wrote about the growing connections between the effective altruism (EA) movement and AI security policy, highlighting ties from top AI startups like Anthropic to influential DC think tanks such as the RAND Corporation. EA focuses on preventing what it sees as catastrophic risks to humanity from future AGI and has built strong links with various government agencies and congressional staff.
Critics argue that EA’s focus on these existential risks diverts attention from current, measurable AI concerns, like bias, misinformation, high-risk applications, and traditional cybersecurity. I’ve been curious to explore the opinions of other AI and policy leaders who are neither part of EA nor its polar opposite, effective accelerationism (e/acc). How do other language model companies view the risk of their model weights falling into the wrong hands? Do DC policymakers understand EA’s influence on AI security?
This issue seems particularly relevant now that Anthropic, known for its EA connections, is publishing research about AI models that evade safety checks, and Congress is concerned about an AI research partnership between NIST and RAND. Additionally, EA made headlines with the firing of OpenAI CEO Sam Altman, whose nonprofit board members had EA ties.
In my recent interviews, I found a mix of deep concern about EA’s billionaire-funded ideological sway over AI security debates in Washington, alongside some acknowledgment of the importance of addressing long-term AI risks.
Effective altruism began with the aim of ‘doing good better’ but is now heavily funded by tech billionaires prioritizing AI security to avert potential catastrophes, particularly in biosecurity. In December, I detailed concerns from Anthropic’s CISO Jason Clinton and two RAND researchers about securing LLM model weights against threats from criminals, terrorists, and nation-states. Clinton emphasized that securing Claude, Anthropic’s LLM, is his top priority, while RAND researcher Sella Nevo suggested AI models might soon be significant to national security, potentially aiding in biological weapon development.
Both companies have EA connections, such as RAND’s CEO Jason Matheny’s involvement with Anthropic’s Long-Term Benefit Trust. Brendan Bordelon of Politico has reported on this issue, highlighting how EA funders are influencing policy circles in Washington.
Nick Frosst, co-founder of Cohere, a competitor to OpenAI and Anthropic, believes large language models don’t pose an existential threat. Cohere focuses on business risks from others accessing their model weights rather than existential concerns. Frosst criticized EA’s focus on AI doomsday scenarios and questioned the movement’s moral justification for wealth accumulation.
Yoav Shoham, co-founder of AI21 Labs, another Anthropic and OpenAI competitor, keeps model weights secret for trade reasons but doesn’t see them as the key enabler for bad actors. He emphasized their commitment to beneficial AI uses and not aligning with EA.
At RAND, Marek Posard criticized philosophical debates like effective altruism and e/acc as distractions for AI policy. While acknowledging real and critical AI concerns, he questioned the utility of these ideological perspectives in practical policy discussions.
Traditional cybersecurity is focused on present-day risks, which might not fully align with EA’s existential concerns. For example, Dan deBeaubien from the SANS Institute emphasized current security risks over potential existential threats, noting limited discussions about effective altruism within his organization.
Other policy experts, like Mark Beall, former head of AI policy at the Department of Defense, emphasize responsible AI acceleration without heavy reliance on EA philosophies. Ian Bremmer of Eurasia Group highlighted present-day AI risks like election disinformation, noting that practical safeguards and responsible usage are essential.
Bremmer criticized the EA focus on end-of-the-world scenarios, arguing that addressing everyday AI impacts on society should be a priority. He believes that while there is a legitimate debate about AI model security, focusing on practical and immediate issues is crucial.
Stay informed with our daily updates on the latest AI news by subscribing to our newsletters.