As the race to build increasingly powerful artificial intelligence systems accelerates, a new concern is emerging even among technology leaders: how far should AI development go, and who should control it?
Microsoft chairman and CEO Satya Nadella has argued that the answer must begin with a simple principle: AI should remain under meaningful human control. If increasingly capable AI systems cannot be controlled by humans or do not deliver broad benefits, he believes they are not worth pursuing.
His comments come amid a growing debate over the risks of frontier AI — highly advanced systems that could eventually operate with greater autonomy and capabilities than today's models.
Nadella's position goes beyond AI safety. He argues that the technology's benefits, control and development should be broadly distributed rather than concentrated among a handful of technology companies.
His vision rests on three broad principles:
Human control: AI systems must remain aligned with human goals and values.
Wider access: The benefits of AI should reach countries, communities and companies across the world.
Distributed power: No small group of technology companies should control the future of AI.
Nadella has also called for greater representation in AI governance, including participation from different countries, industries and academic institutions.
One of the more significant aspects of Nadella's argument concerns businesses and their proprietary knowledge.
Companies increasingly use AI to process internal data, customer information, operational experience and specialised expertise. Nadella believes they should not become permanently dependent on a single AI provider to make use of that knowledge.
He has argued that organisations should be able to build their own "continuous learning loop" — continuously improving AI systems using their own data, feedback and experience.
For businesses, this could mean:
retaining greater control over proprietary knowledge;
continuously training and improving AI systems using internal data;
reducing dependence on a single model provider;
having greater control over how business knowledge is incorporated into AI models; and
maintaining flexibility between proprietary and open-source AI models.
This is becoming an increasingly important business issue as companies embed AI deeper into their operations.
Nadella has backed efforts for deliberate pacing in the development of frontier AI, particularly where safety and alignment are concerned.
AI alignment broadly refers to making sure advanced AI systems behave in accordance with human objectives, values and safety requirements.
He has also supported ideas such as independent or embedded evaluators that can assess AI systems and identify potential risks.
The debate intensified after Anthropic CEO Dario Amodei called for the AI industry to "pace the frontier" and give safety measures more time to catch up with rapidly advancing capabilities.
Amodei has proposed stronger independent evaluation of AI systems, including giving external safety evaluators access comparable to that of employees.
The proposal has attracted support from two other major players in the AI race.
OpenAI CEO Sam Altman backed the idea of pacing frontier AI development and supported independent evaluators having employee-level access.
xAI founder Elon Musk also endorsed Amodei's position.
The support is notable because OpenAI and xAI are themselves competing to develop increasingly powerful AI systems.
It highlights a growing realisation within the industry that technological progress and safety cannot be treated as separate questions.
Nadella's warning is also about who gets to decide the future of AI.
He has argued that AI governance should not be controlled by a handful of technology companies. Instead, it should involve a much wider ecosystem of:
technology companies;
governments and countries;
universities and researchers;
businesses and users; and
developers working on both closed and open-source models.
The objective, he suggests, is to ensure that AI remains a technology with broad participation rather than one whose capabilities and direction are determined by a small group of companies.
Nadella said Microsoft's AI strategy would focus on broad access and choice across the AI stack, while giving enterprises greater control over their own learning loops and models.
Microsoft also plans to establish a "Code of Conduct" for its first-party MAI models, with the framework to be made available for public consultation.
The larger message from Nadella is clear: the AI race cannot be measured solely by how powerful the next model becomes.
For businesses and society, the more important questions may be who controls that power, who benefits from it and whether humans can still determine how it is used.
As AI moves from being a productivity tool to becoming an increasingly autonomous business technology, those questions could prove as important as the technology itself.
