Coalition for Health AI Unveils Registry to Standardize AI Tool Evaluation
A public-private partnership dedicated to establishing standards for artificial intelligence (AI) in healthcare has launched a new registry. This platform is designed to provide hospitals and developers with a centralized resource for accessing and sharing information on AI tools.
The Coalition for Health AI (CHAI), in collaboration with health system members like Providence, Cleveland Clinic, and Kaiser Permanente, introduced the model card registry. The registry aims to simplify the evaluation and comparison of validated AI products for healthcare companies.
According to CHAI, AI companies and other developers can submit their tools to the registry regardless of their CHAI membership status. The platform is currently free to use, offering access to model cards and other resources, though future paywalled features are possible.
AI’s Growing Role in Healthcare
AI is gradually becoming more integrated into the healthcare sector. Providers are increasingly using algorithms for tasks such as automating clinical note-taking, patient interactions, streamlining staff scheduling, and other administrative functions.
A survey from HIMSS, conducted in the fall, revealed that 86% of respondents already use AI in their medical organizations, primarily for administrative purposes.
There is growing interest in expanded clinical applications of AI, including tailoring treatment plans and supporting clinicians with diagnostic processes, though adoption remains relatively new.
One of the primary challenges in implementing AI is the industry’s rapid pace of change. Healthcare organizations must evaluate and select from various AI tools that have widely varying claims.
Healthcare providers must guarantee ethical AI implementation and patient safety, as concerns about rushing to adopt AI at a pace that outstrips oversight capacities have grown. CHAI, along with other groups, hopes to create standards for the responsible use of AI.
The new registry standardizes information about AI tools using model cards, which document an AI model’s development, capabilities, and limitations in a standard format.
CHAI states that model cards are intended to act like a nutrition label for food. The registry, developed with Avanade, a joint venture between Microsoft and Accenture, allows hospitals to access centrally located model cards while giving AI tool developers a centralized, accessible venue in which to make claims.
“The model cards play a crucial role in the AI governance process,” said Elisabeth Garwood, associate chief medical informatics officer at UMass Memorial Health in Massachusetts, in a statement. “By consolidating information in an easily digestible format, facilitating product comparisons, and providing standardized data that meets the needs of various stakeholders, from radiologists to legal teams.”
CHAI released its model card template for public comment last year. The commentary was varied, with stakeholders requesting to amend the information on the card based on their industry.
CHAI has not yet adopted these changes, but the template isn’t final. The consortium is working to ensure the model card includes appropriate fields for generative AI products, which are more difficult to oversee than predictive models. CHAI is also focused on transforming the model registry into a tool that delivers users the right information at the right time.
A spokesperson declared this will require “in-depth testing with a wide variety of stakeholders and organizations. This means testing in the actual workflow.”
Since it was founded in 2021, the consortium has expanded to contain over 1,300 member organizations. Some members include big tech companies such as Microsoft, Google, and Amazon. CHAI, which included members of the federal government during the Biden administration, aims to create a network of quality assurance labs to evaluate AI models and develop best practices for deployment.
CHAI is one group led by massive tech companies, AI developers, and hospitals that aims to address a void created by inconsistent federal government involvement regarding federal AI oversight. President Donald Trump previously overturned a Biden-era guidance calling on agencies to create schemata for overseeing AI, including within healthcare, as well as promoting innovation in the U.S. in response to the success of China’s AI model.
Other standards groups include the Trustworthy & Responsible AI Network (TRAIN), also supported by Microsoft. Nvidia, a major hardware builder essential for AI applications, is also collaborating with Aidoc, a clinical AI company, to release its standards blueprint this year.
