New Innovation Models in Medical AI


In recent years, scientists and researchers have devoted considerable resources to developing medical artificial intelligence (AI) technologies. Many of these technologies—particularly those that resemble traditional medical devices in their functions—have received substantial attention in the legal and policy literature. But other types of novel AI technologies, such as those related to quality improvement and optimizing use of scarce facilities, have been largely absent from the discussion thus far. These AI innovations have the potential to shed light on important aspects of health innovation policy. First, these AI innovations interact less with the legal regimes that scholars traditionally conceive of as shaping medical innovation: patent law, FDA regulation, and health insurance reimbursement. Second, and perhaps related, a different set of innovation stakeholders, including health systems and insurers, are conducting their own research and development in these areas for their own use without waiting for commercial product developers to innovate for them. The activities of these innovators have implications for health innovation policy and scholarship. Perhaps most notably, data possession and control play a larger role in determining capacity to innovate in this space, while the ability to satisfy the quality standards of regulators and payers plays a smaller role relative to more familiar biomedical innovations such as new drugs and devices.


Innovation in medical AI is exploding. Every week sees new research papers presenting new algorithms, new companies launching new products, and new possibilities for change. AI products promise to recognize and diagnose skin cancer, to identify eye disease, to find kidney stones, to locate brain hemorrhages, and to quickly detect COVID-19, among many other possibilities.[2] These technologies are likely to change the practice of medicine by increasing the capabilities of care providers in many areas. Products like these also fit—if somewhat uncomfortably[3]—into a capacious understanding of what new medical technology looks like and how we expect it to be regulated. But these are not the only AI products with the potential to transform medicine.

Other AI innovations look quite unlike typical medical devices, yet they also have the potential to transform health care in different ways.[4] A seemingly mundane example is AI-powered scheduling software, which predicts the ebb and flow of patients within the health care system and allocates staff to most effectively meet those patients’ needs. Such products do not directly diagnose or treat patients, but they could increase the capacity of a stretched system and thereby save lives. Other products improve quality of care by predicting the likelihood that a patient will be readmitted to the hospital within a month (so that health care providers can work with patients to prevent that undesirable outcome) or by identifying the risk of a patient developing sepsis (so that rapid-response teams can intervene early). These functions are extremely valuable to the health care system, and all are amenable to AI assistance.

For these forms of AI innovation, however, the traditional policy levers that shape much biomedical innovation—patents, FDA regulation, and insurance reimbursement[5]—play more uncertain and attenuated roles. Although many innovators are actively pursuing patents, the patentability of medical AI under U.S. law is unclear, making it risky to enforce AI patents that a court might hold invalid. Patents may also be less important to would-be innovators because AI innovations are often easy to protect via trade secrecy. Some of these technologies may get less scrutiny from FDA, perhaps because they do not fit within the statutory definition of medical devices, or perhaps because they fall within categories for which FDA has traditionally exercised discretion not to enforce its authorities. And insurance reimbursement, which normally helps both to drive the development of medical technology and to provide some quality-related oversight, plays little role here, as these products are typically not directly reimbursable. The usual incentives of insurance reimbursements or patent law exclusivity are thus lower for these forms of innovation, but barriers to entry from FDA or insurer oversight may be lower as well. To be sure, some medical AI innovators do seek patents and FDA approval or clearance. Nonetheless, medical AI innovation faces a substantially different legal landscape than more conventional biomedical innovation, such as the development of physical devices or drugs.[6]

Within this landscape, innovation by end-users of medical AI is flourishing. Health systems (including individual academic medical centers and hospitals) and insurers are not only developing and using AI technologies themselves but also setting up in-house venture capital funds to invest in AI startups. Health systems and insurers have different incentives than conventional biomedical innovators (such as drug and device manufacturers). Their primary purpose for innovating is not to sell innovative products to customers. Instead, they are developing innovative AI tools to enhance their main business of providing, insuring, or facilitating health care. In the theoretical model pioneered by Eric von Hippel,[7] they are “user innovators” rather than seller innovators.[8] They benefit directly from using their innovations without having to sell or license them to others (though they may do both). User innovators are more likely to focus on their own specific needs and circumstances, creating more customized products for their own use, while seller innovators are more likely to produce standardized products designed for sale to a broader market of users.[9]

Of course, users are not the only innovators of medical AI. Large technology companies are developing AI-powered health software for sale to users, as are small startups. And the IT infrastructure providers of health care, the makers of electronic health record (EHR) software, are themselves developing AI algorithms and incorporating them into EHR products. But the innovation incentives for commercial product developers are somewhat more familiar and not our focus here.

The rise of user innovation in biomedical AI has several implications for policymakers. First, it is worth considering that the different legal landscape in this setting may be making room for different kinds of innovators to develop different forms of innovation. Just as the ordinarily robust legal regimes that provide patents, FDA regulation, and insurance reimbursement for new technologies shape biomedical innovation in drugs and physical devices,[10] the smaller roles these regimes play for medical AI may shape the different forms of innovation that we observe in this space, perhaps making more room for user innovation. Second, the availability and control of data confer a significant comparative advantage on some innovators in this field, including large institutional user innovators. AI is easier to develop in-house for health systems or insurers with their own large stocks of patient health information. Smaller institutions, or commercial firms without access to such data, may find it harder to compete. Third, a proliferation of biomedical user innovators brings challenges as well as opportunities. AI innovations tailored to one institution’s needs and circumstances may not be suitable for other potential users facing different needs and circumstances.[11] Even larger institutional datasets are limited in scope, limiting the power and generalizability of AI solutions based on those datasets. Problems of error, overfitting, or data biases might go unrecognized without effective oversight from FDA or insurers. These effects have broader impacts on the quality, cost, and equity of medical AI more generally.

The rest of this Article proceeds in three Parts. Part II canvasses the landscape of user innovation in medical AI and describes the novel innovators involved, focusing on the roles and incentives of health systems and health insurers. Part III looks to the primary regimes that scholars have generally recognized as shaping biomedical innovation—patent law, FDA oversight, and insurance reimbursement—and explains how their role is diminished or uncertain for these technologies. Part IV addresses the implications of these analyses, including concerns around the availability of data, the customization of local solutions to local problems, and risks of difficult-to-detect quality concerns. A few brief thoughts conclude.

I. New Technologies and New Innovators

AI powers a proliferating set of new medical technologies. Some AI tools are directly involved in patient care, such as systems that diagnose medical issues or monitor patients for signs of medical problems that can be aided by early intervention. Some function more in the background, such as algorithms to predict the likelihood of future adverse outcomes. Still, others are even further removed from the point of patient care, monitoring and shaping the flow of patients or providers across a hospital to improve system efficiency or to increase the volume of care provided. Each of these avenues has the potential to impact the health care landscape and the experience of patient care.

The technologies we consider here fall largely outside the scope of existing policy and legal scholarship on medical AI. That small but growing body of scholarship has considered legal aspects of commercially developed AI-driven products that directly drive or inform patient care and that pass through FDA’s traditional review process.[12] An example is IDx-DR, a software program that diagnoses more-than-mild diabetic retinopathy based on images of the base of the retina. IDx-DR was cleared in 2018 by FDA as a Class II medical device and has since been sold commercially and implemented at sites around the country.[13] FDA has cleared dozens of medical devices[14] that rely on AI to perform a function like classification, diagnosis, or risk prediction. These products, while important, are not our focus here. Instead, we consider the vast breadth of AI-powered medical technology that arises outside the typical product development path.

The innovators we consider in this project—health systems and insurers—are user innovators. Although much of the user innovation literature in recent years has focused on individuals and hobbyists rather than institutions and professional researchers, the defining characteristic of user innovators is not their identity or status, but their functional relationship to the innovation at hand. They innovate to address their own immediate problems or to adapt available technologies to work for their purposes, typically because commercially available products are inadequate to meet those needs.[15] This distinguishes these innovators from companies specializing in the development and sale of cutting-edge health care technologies such as pharmaceuticals or medical devices for sale to others.[16] Those companies aim to supply broader markets with new therapeutic products after shepherding them through costly premarket testing, navigating complex federal bureaucracies to secure intellectual property rights, FDA clearance or approval to gain market access, and showing that they qualify for insurance reimbursement to ensure commercial success. The cost, risk, and time needed to bring to market a new pharmaceutical[17] or medical device[18] limit the companies that are able to succeed in this complex environment. Pharmaceutical and medical device companies specialize in dealing with these regulatory structures and are shaped by these legal and financial dynamics.

The medical AI innovation context we examine here is quite different. FDA regulation, patents, and insurance payments still matter, but the costs of navigating the legal landscape are less daunting, and the rewards more uncertain (as we will discuss infra, in Part III). Innovation depends less on the ability to conduct clinical trials than on access to large volumes of data collected in the course of clinical care. In this environment, health systems and insurers have begun to play more significant roles in medical AI innovation.

A. Health Systems

Health systems[19] play an integral role in the delivery of health care services as well as in health care research. However, they have typically not been a focus of legal academic scholarship about the development of new health care technology products.[20] To be sure, the medical literature recognizes that hospitals—particularly academic medical centers—serve as research sites in the clinical trials process, providing patients for enrollment in trials to test the safety and effectiveness of a candidate drug or device.[21] But in these contexts, the outside product manufacturer may be the party in control of the research, rather than the hospital or health system itself.

The dynamics are different in the context of AI technologies. Health systems themselves have played a larger role in driving the development of a wide range of innovative AI products, but with different incentives than those of the product developing firms that are the focus of much of the scholarly literature. In the AI context, health systems are less concerned with the ability to obtain patents, the prospect of securing insurance reimbursement for their new products, or the need to traverse the FDA clearance or approval process.

Instead, they seek to reduce their own costs, increase clinical volume and revenue, improve quality, and satisfy genuine scientific curiosity. Importantly, though, health systems may be unable to meet these goals with one-size-fits-all AI products. Different health systems may not only weigh these goals differently, but also have different patient populations and different structural constraints that they must accommodate through customization and training of AI products. Hospitals of different sizes, with different specialties, or with different seasonal patient volumes will need to develop different ways of managing capacity strain and provider staffing, for example.

These features make the development of AI models for health systems (sometimes in collaboration with external firms) a good candidate for user innovation.[22] Some problems differ across institutions enough that off-the-shelf models have limited value; AI allows health systems to address such challenges. Health systems, in many cases, have sufficient resources to develop their own models (or at least to contribute substantially to the development of such models).[23] Medical AI tools trained on their own data offer health systems opportunities to improve their own operations at a reasonable cost. Use of their own data both limits the costs of innovation and ensures that the results are targeted to their own needs and circumstances.

Health systems face a number of motivations to pursue this innovation. They feel pressure to compete on quality, especially in light of HHS’ imposition of financial penalties for particular types of complications—and financial bonuses for others.[24] High-quality medical care can be difficult to deliver consistently because of differences among patients that are difficult to observe. Ideally, caregivers would monitor patients continuously along many dimensions, using all available information to choose exactly the right intervention for each patient at exactly the right time. This would allow caregivers to treat patients quickly and effectively while also avoiding unnecessary treatment. But it is a challenging goal. Constant monitoring is resource-intensive, and accurate analysis requires skill and knowledge. AI can help by monitoring patients and predicting or quickly identifying adverse events in time for intervention.[25] Because AI systems base their predictions on huge amounts of data, the underlying rationale may be opaque to human observers.[26] Many health systems have begun to develop AI tools that will assist physicians in lowering their institutions’ rates of different types of adverse events.

At the University of Michigan, researchers developed a predictor for the risk of infection with Clostridium difficile (C. diff.), a bacterium that infects hundreds of thousands of patients per year in health care settings.[27] C. diff. can be deadly or debilitating, and has become increasingly resistant to antibiotic treatment.[28] The Michigan team used electronic health record (EHR) data from many thousands of patients to develop the predictive tool, which now makes daily predictions at Michigan Medicine to identify patients at high risk of infection for closer monitoring. The predictor is based on hundreds of EHR-derived variables, some of which accord with prior clinical understanding (e.g., high respiratory rate) but most of which do not.[29] The predictor can also be used to drive systematic improvements, such as isolating particularly vulnerable patients from the rest of the health system’s population to protect them from infection.

Other health systems have developed their own C. diff. prediction tools trained on their own data.[30] Some have tried to create one-size-fits-all models that work for all institutions, but have encountered difficulties based on differences between health systems.[31] The Michigan team collaborated with researchers from Mass General Hospital on an intermediate approach: a generalizable method that can be used to develop models that fit the particular health systems that will use them.[32]

Duke University has developed and implemented its Sepsis Watch system to monitor patients for sepsis. Sepsis is a serious and often fatal condition in which the body’s inflammatory response to an infection goes into overdrive. It can quickly become fatal and kills about 270,000 patients annually in the United States. Duke’s AI system, trained on EHR data, makes real-time predictions about patients’ risk of sepsis and alerts a rapid response team to intervene early and catch sepsis in its early stages. The system is relatively opaque because “[c]linical leaders . . . were willing to trade-off model interpretability for performance gains.”[33] Researchers did not prioritize model interpretability, because sepsis may have many causes, and treatment does not depend on which of those causes is present. Duke developed the system in its main hospital and has since rolled it out—with substantial effort and adaptation—at its two other, smaller hospitals.[34] During adoption, Duke explicitly promoted the system to providers “as a home-grown solution to an important problem within the hospital,”[35]—though Duke has since licensed Cohere Med to develop the unpatented system for wider use in other settings.[36]

Health systems are also acutely interested in patient readmission—that is, the likelihood that a discharged patient will be readmitted to a hospital within a given time frame (typically thirty days). Thirty-day readmission rate is a marker of care quality, and something health systems try to minimize.[37] Readmission within a short time frame is a sign that something has gone wrong with the patient’s care: perhaps the patient’s issues were not properly resolved, or the patient should not have been discharged yet. Multiple health systems have developed their own AI-powered tools to identify patients at high risk of readmission to target them for intervention (such as assigning a nurse to coordinate their outpatient care). Researchers at the University of Texas Southwestern hospital in Dallas developed a thirty-day readmission model, which they externally validated in seven large hospitals.[38] UT Southwestern has since spun out the model to the private firm Pieces, which now offers it as a part of its “Pieces Predict” commercial product.[39]

Yet another example: Intermountain Healthcare has partnered with an external firm to develop better ways of managing patients with chronic kidney disease, with the goal of reducing hospitalizations and improving outcomes.[40] Tools like these could enable health systems to provide better care at a lower cost—while maintaining existing levels of service provision and insurance reimbursement.[41]

Health systems generally seek to increase clinical volume and revenue where possible. They therefore have incentives to develop AI tools that can, for example, assist physicians in completing procedures more quickly or identify additional patients who would benefit from further services. One such example comes from Cedars-Sinai in Los Angeles,[42] which has developed an AI tool to reduce capacity strain on the system. Capacity strain may lead to crowded ERs, delays or cancellations of surgeries, unnecessary readmissions, and provider burnout. By predicting more accurately the hospital’s patient census, the Cedars-Sinai AI tool aims to decrease treatment delays, improve staff schedules (including reducing the need to pay overtime), and increase admissions, while avoiding overcrowding.

Other examples focus on resource allocation and efficiency, decreasing the resources needed for care and, perhaps simultaneously, increasing the volume of care provided with existing resources. Duke is using AI to optimize bed flow—that is, the movement of patients between different hospital units during different time periods after admission.[43] After the COVID-19-related shutdown of elective surgery, Duke also turned to AI to prioritize the most important elective surgeries[44] and to reduce costs by allowing low-risk patients to be “admitted” to the hospital while staying home and receiving in-person visits, telemedical care, and remote monitoring.[45]

Health systems also aim to reduce different types of costs. In particular, health systems seek to reduce back-end costs related to coding, billing, and transacting with third parties (such as insurers or regulators), as these costs do not themselves either serve a direct patient care mission or garner reimbursement for the hospital.[46] They represent administrative frictions that cannot be eliminated entirely but are often far higher in the highly-fragmented U.S. health care system (where providers must develop the infrastructure to contract with and bill a variety of different insurers) than in other, less-fragmented health care systems.[47] Some AI innovations assist health systems in reducing these back-end costs. For example, one health system uses AI to analyze physician visit notes for reimbursable events that were not coded for reimbursement and flags those events for human review.[48] This system not only increases revenue for already-provided care; it also decreases the cost of human review devoted to billing.[49]

Although health systems are businesses subject to standard corporate financial incentives,[50] scientific curiosity often plays a motivating role as well. Academic medical centers, in particular, perform clinical research in addition to providing patient care.[51] For academic medical centers, advancing knowledge is a part of the institutional mission—and given the substantial grant funds available for biomedical research, including in the medical AI field,[52] innovation may also have financial implications.[53] Academic medical centers have been at the forefront of new AI research that might be less attractive to commercial firms focused on lucrative sales of patented products and insurance reimbursement. Some of this research more closely resembles traditional basic research into the drivers and progression of certain conditions[54] than it resembles the more applied innovations being developed for the prevention of sepsis or readmissions described above. Many of these basic-research initiatives are government-funded and use techniques of artificial intelligence and machine learning to gain a greater understanding of particularly complex conditions, such as Alzheimer’s Disease or brain genomics more generally.[55]

The smaller role of the traditional innovation policy levers discussed in Part III may, if anything, enhance the comparative advantage of user innovators and make it easier for health systems to engage in this broad range of AI innovation. Health systems seeking to develop AI technologies can do so with far less financial investment than companies developing traditional medical products, for several reasons. Rather than paying for costly data collection through clinical trials or acquiring data held by other institutions, they can repurpose data that they have already created in the form of health care records of clinical care.[56] Moreover, for reasons more fully explained in Section III.A, they are much less likely to go through a costly FDA review process for their AI technologies.

As a result of these lower development costs, health systems may not need substantial, standalone reimbursements for these innovations and can instead recoup their investments directly through their own use to reduce costs, to increase the volume of care they deliver and bill for, or to improve quality metrics. In other cases, health systems are covering their investment expenses by obtaining grants or prize awards (most notably but not only from the federal government).[57] In still other situations, as noted above, health systems may develop these AI tools in collaboration with outside firms, contributing their valuable patient health data for the company’s use.[58] Patent protection may be less important for these AI technologies because of the difficulty for competitors of reproducing technologies that rely on access to confidential data sets and use opaque algorithms.[59] Finally, patent protection may be superfluous for truly bespoke innovations that are customized to meet the needs of the innovator, because they may not be suitable for use by competitors.

B. Insurers

Health insurers are longtime stakeholders in the delivery and coverage of health care services. Insurers themselves are a varied group with diverging interests. Even putting aside the role of the federal government as an insurer, providing coverage for more than 100 million Americans through Medicare and Medicaid alone,[60] private insurers play a range of different roles in health care delivery and coverage. Insurance companies may provide insurance for businesses that offer health benefits to their employees, or may serve as third-party administrators that process insurance claims for employers who self-insure coverage for their employees.[61] Insurance companies may offer fee-for-service plans, reimbursing providers for each service they provide, or they may use a managed care model, requiring providers to work within more tightly specified budgets.[62] Private insurance firms even play a large role in the deployment of the Medicare and Medicaid programs, as twenty-four million seniors now purchase privately-run Medicare Advantage plans[63] and nearly fifty-four million Medicaid enrollees have their coverage provided by comprehensive Managed Care Organizations through private insurers.[64]

In general, however, insurers have not featured prominently in discussions of the process of innovation into new health care technologies.[65] But the role of insurers has changed for new AI technologies. Like the hospitals and health systems described above, insurers have incentives to reduce care costs and increase efficiency that has driven them to invest in the development of certain types of new AI products.[66] Insurers can also be user innovators seeking to develop customized products for use in their own operations.

Insurers have financial motivations to reduce care costs. If an insurer has budgeted a particular amount of money for the care of each beneficiary each year, costs that exceed that projected budget will often be borne by the insurer, not by an employer or by the patient. Insurers thus have incentives to discourage patients from seeking unnecessary care, where possible. For example, an insurer might prefer that a patient see a primary care doctor or visit an urgent care clinic for non-emergency care, rather than going to a (more expensive) emergency room.[67] Many insurers have developed non-AI initiatives designed to help patients figure out what type of care might be right for them, such as providing 24-hour triage nurse lines.[68]

Insurers are also working on AI-based products that help both doctors and patients to make triage decisions. Highmark, affiliated with Blue Cross & Blue Shield, aims to use AI tools both to prevent the onset of chronic conditions and to treat them more effectively.[69] Highmark’s recent development partnership with Google Cloud will allow Highmark to contribute its patient data to the collaboration, with the goal of benefiting from Google Cloud’s AI expertise.[70] At least some of these AI tools—such as one developed by Optum, a division of UnitedHealth, to better manage patients with chronic conditions—have resulted in disturbing racially disparate impacts on patients,[71] a topic to which we return in Part V.

Closely related to reducing the costs of care, insurers are motivated to increase efficiency in the reimbursement process. Just as hospitals and health systems must develop the infrastructure to contract with and bill many different insurers for the care they provide to their patients, insurers must develop the infrastructure to work with many different health care providers and to manage the claim review process. Insurers are working to develop AI-based technologies for reducing the administrative costs and frictions of these interactions. For example, Optum has also developed AI models to help review provider claims.[72] Overall, the innovation incentives of insurers seem more tightly tied to reducing costs along multiple dimensions, both care-based and administrative.

C. Venture Capital Investments

In addition to developing AI tools either in-house or in partnerships with external firms, both health systems and insurers are also pursuing new ways to provide venture capital funding for outside innovators.[73] At least some of these VC-funded efforts are designed to produce novel AI-based technologies. Many of them also fall into the above categories—for instance, Cigna Ventures has invested in a company using AI to target precision medicine efforts, aiming to target treatments to particular patients.[74] UnitedHealth’s Optum Ventures has devoted a portion of its $600 million venture fund to Mindstrong Health,[75] which seeks to deliver mental health care virtually in a way that functions to “lower[] the inpatient readmission rate” and “ER admission rate.”[76]

Health systems also fund innovations that come directly from their own internal work, but which may need further external development. Several of these funds are sponsored by large, well-known health systems—such as the Mayo Clinic[77] and the Cleveland Clinic[78]—but many smaller health systems have funds as well. Providence Ventures, the venture capital fund of Seattle-based Providence Health & Services, plans to invest $150 million in new IT products “designed to improve care coordination, patient engagement, data analytics” and other priorities.[79]

When these innovators are working to meet their own needs rather than to sell to a commercial market, they respond differently to the set of legal levers that policymakers often use to manage innovation—a subject to which we now turn. These innovators are relatively undeterred by the uncertainty of patent protection for their innovations, and if they are not selling a product, they have no need to reassure purchasers that the high price of their innovations will be covered by insurance. At the same time, in some cases, these AI innovators may avoid the lengthy, risky FDA review process before putting their products to use, making the costs of developing these AI products far lower than the costs of developing conventional therapeutic products for commercial sale. These altered dynamics help explain both why different innovation stakeholders have emerged as significant movers in medical AI innovation and also how those stakeholders’ incentives are shaped.

II. Diminished Legal Regimes: Quality Oversight and Incentives

Three major legal regimes that shape other kinds of biomedical innovation are less robust in the context of these new AI technologies than they are for other biomedical innovation. Most of these AI tools are subject to substantially less rigorous FDA scrutiny than are traditional new prescription drugs or medical devices, either as a matter of statutory constraint or as a matter of FDA’s exercise of enforcement discretion. Patent incentives are less reliable for several reasons, including limitations on patent-eligible subject matter under U.S. law and difficulties complying with patent law disclosure requirements for algorithms that are opaque and constantly changing. Patents may also be less important because of the effectiveness of trade secrecy for these innovations or because these customized products are not suitable for other users and therefore less vulnerable to free riding. And because most of the AI technologies involved are not reimbursable by insurers, insurance coverage determinations fail to supply either direct incentives or an independent source of quality oversight. The relative weakness of these regimes for these AI technologies, on one hand, reduces the legal incentives that usually motivate seller innovators, but on the other hand, reduces barriers to entry. At the same time, the weakness of these regimes limits the levers available to policymakers seeking to shape the development of this burgeoning set of AI tools. Other mechanisms are available—grants or prizes could drive development, and tort law or state medical boards could provide oversight—but the traditional policy levers are harder to pull.

A. FDA Regulation

FDA performs a critical technology oversight role under the Food, Drug & Cosmetic Act (FDCA)[80] before many new biomedical technologies may be introduced in commerce. Although regulation adds to the costs of developing these products, it also promotes innovation in at least two ways. First, by demanding that data from clinical trials of new products be collected and submitted to FDA as a condition for premarket approval or clearance, regulation motivates innovating firms to invest in a costly and socially valuable form of R&D. Second, by imposing regulatory entry barriers on other firms before they can market competing versions of successful new technologies, regulation gives innovators a head start before they face price-lowering competition.[81]

The regulatory landscape appears to be quite different for many technologies described in this Article. Health systems and insurers are routinely developing and implementing AI systems that shift the way care is provided, whether directly or indirectly, without seeking FDA clearance or approval—indeed, as of this writing, to our knowledge, no FDA cleared or approved AI devices were sponsored by health systems, hospitals, academic medical centers, or insurers.[82] And yet, such user innovators regularly deploy their own AI-based systems, as described above. What explains this difference?

Leaving aside the possibility that some innovators may be flouting FDA requirements, there are several good reasons that the agency keeps a lower profile here. Some of the technologies considered in this paper are likely beyond the reach of FDA’s regulatory authority. Others may be within FDA’s authority, but it may decline to regulate them as a matter of enforcement discretion because of its current perception that they do not present much risk to patients.[83] FDA is actively reevaluating its regulatory approach to AI and machine learning functions that are intended for use in the care, mitigation, treatment, or prevention of disease, leaving some uncertainty as to what future regulation will look like.

We do not mean to suggest that the agency is absent—even where it leaves unregulated spaces, the contours of those spaces shape the actions of health systems and insurers in developing the technologies noted above. Nevertheless, the overall picture is one of lower regulatory hurdles for medical AI, particularly when it is developed and deployed by user innovators. This is important because high regulatory costs might otherwise favor seller innovators who can spread those costs over a larger market of consumers of standardized products and disadvantage user innovators who develop customized products for their own use.[84]

1. The Limits of FDA’s Regulatory Authority

FDA has never had comprehensive authority to regulate all new medical technologies, and much biomedical innovation has routinely happened in the course of activities that are beyond FDA’s reach. By long tradition, FDA does not regulate the practice of medicine, including innovative new uses by physicians of products that were previously approved or cleared as safe and effective for other purposes.[85] Health care providers have always played an important role in biomedical innovation as they learn by doing,[86] giving critical feedback and suggestions for improvements to firms that develop regulated products as well as continuously improving unregulated technologies such as surgical techniques.[87]

The FDCA only applies to products that are introduced, delivered, or received in interstate commerce, an important limitation that may exclude many user innovations.[88] Many of the technologies that we consider in this paper are developed and used within institutions like health systems that do not sell them to others or otherwise make them available in commerce.[89] As a result, information technology products developed within a health care institution without the use of components derived from commerce, and used only internally to analyze the institution’s own data, are arguably beyond the constitutional and statutory limits of FDA regulation. Lawyers have certainly made this argument regarding FDA’s authority to regulate laboratory-developed diagnostic tests, which are also developed and deployed within a particular health care institution.[90] However, courts have also upheld FDA’s regulatory authority in arguably similar circumstances.[91]

Another limitation on FDA’s authority in this context is that not all of the AI technologies considered in this paper are likely to fit within the broad statutory definition of “device”:

an instrument, apparatus, implement, machine, contrivance, implant, in vitro reagent, or other similar or related article, including any component, part, or accessory, which is … intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease, in man or other animals . . . .[92]

This definition, which covers a broad range of health care products from simple bandages and tongue depressors to complex cardiac pacemakers,[93] nonetheless might be read to exclude software to improve the efficiency of health system staffing operations. On the other hand, an algorithm that predicts which patients are at heightened risk of developing a C. diff. infection and selects some patients for closer monitoring seems clearly to be “intended for use . . . in the care, mitigation, treatment, or prevention of disease in man.”[94]

Congress further limited the definition of device in the 21st Century Cures Act (Cures Act),[95] to exclude five specified “software functions,”[96] while generally preserving FDA’s traditional authority to regulate products intended for use in the diagnosis, treatment, or prevention of disease.[97] Some of the functions recited in these exclusions, such as “administrative support of a healthcare facility,” would not likely have been regulated as medical devices even prior to the Cures Act because they fall outside the “intended use” limitation in the statute.[98] Others, such as maintaining or encouraging a healthy lifestyle, involve functions of low enough risk that FDA previously indicated it would decline to regulate them as a matter of enforcement discretion.[99] Although these functions are now presumptively excluded from the statutory definition of device, Congress gave FDA authority to regulate them as devices if it makes a finding “that use of such software function would be reasonably likely to have serious adverse health consequences.”[100]

Two exclusions, described in subsections (D) and (E) of the Cures Act software provisions, potentially curtail regulation of traditional software that might otherwise have been covered by the broad statutory definition of device, but they generally leave intact regulatory authority over more complex medical AI. Subsection (D) excludes software functions that transfer, store, convert formats, or display data, “unless such function is intended to interpret or analyze” the data, in which case it remains subject to regulation as a medical device.[101] Under FDA’s interpretation, this provision allows FDA to regulate software that allocates health system resources to those patients with the most urgent needs:

For example, if a software function is intended to prioritize patients in an Intensive Care Unit based on their clinical status, then this function is intended to interpret or analyze device data, results and findings and is, therefore, not excluded from the definition of device. . . .[102]

Many forms of AI technology described in this article would seem to remain subject to regulation under this interpretation.

Subsection (E) provides a potentially broader exclusion for clinical decision support (CDS) software for the use of health care professionals, but it appears not to apply to opaque recommendations derived from complex AI algorithms.[103] This exclusion covers some software functions that analyze data and that provide recommendations to a health care professional about prevention, diagnosis, or treatment of a disease or condition,[104] but only if it is intended to be sufficiently transparent to enable a health care professional “to independently review the basis for such recommendations . . . so that it is not the intent that such health care professional rely primarily on any of such recommendations to make a clinical diagnosis or treatment decision regarding an individual patient.”[105] As FDA explains in recent draft guidance, this criterion requires disclosure of underlying data and the logic or rationale used by an algorithm in making a recommendation to qualify for exclusion:

In order to describe the basis for a recommendation, regardless of the complexity of the software and whether or not it is proprietary, the software developer should describe the underlying data used to develop the algorithm and should include plain language descriptions of the logic or rationale used by an algorithm to render a recommendation. . . . A practitioner would be unable to independently evaluate the basis of a recommendation, and therefore would be primarily relying upon it, if the recommendation were based on information whose meaning could not be expected to be independently understood by the intended HCP user (e.g., the inputs used to generate the recommendation are not identified).[106]

At least some health care systems are working to align their AI products with this criterion to avoid regulation. In the case of Sepsis Watch, discussed above in Part II, “[c]linicians were instructed to put the model output into context with other relevant information to confirm or dismiss a sepsis diagnosis. The machine learning model did not drive clinical care in a standalone manner.”[107] The developers behind the tool reported that they “worked closely with regulatory officials to ensure that Sepsis Watch qualified as CDS and was not a diagnostic medical device.”[108]

Commentators have criticized this statutory criterion and FDA’s interpretations.[109] The criterion may be difficult or impossible to satisfy for sophisticated AI software that continuously learns from new data and makes recommendations based on ever-changing algorithms that are opaque to users. Even when transparency is technically possible, it may require disclosure of valuable proprietary data and algorithms, thus forcing innovators to choose between avoiding regulation and preserving trade secrecy.

2. Enforcement Discretion and Its Limits

In addition to statutory limits on what FDA can regulate, FDA sometimes exercises discretion to relieve innovators from the burdens of regulation for relatively small-scale activities. For example, when Congress gave FDA authority to regulate in vitro diagnostic devices as medical devices in the Medical Device Amendments of 1976,[110] FDA exercised discretion to refrain from enforcement for laboratory developed tests (LDTs) that are designed, manufactured, and used within a single laboratory.[111] This limitation significantly reduces costs for some user innovators, although at the social cost of reducing regulatory oversight of health and safety. Initially these laboratories were small and local, but as the industry and technology evolved, the entities taking advantage of enforcement discretion became larger and provided testing services on a national and even international scale.[112]

Eventually FDA issued draft guidance proposing to exercise greater oversight of some LDTs under a risk-based approach that would increase oversight as necessary to protect patient safety.[113] FDA decided against issuing final guidance for the regulation of LDTs in the final days of the Obama Administration[114] (though the laboratories themselves remain subject to regulation by the Centers for Medicare and Medicaid Services (CMS) under a different statute).[115] But the absence of binding guidance does not compel FDA to continue a policy of enforcement discretion. Meanwhile, some laboratories have sought premarket approval or clearance for LDTs, perhaps to signal quality or to secure insurance coverage for their tests.[116]

FDA recognized the complex effects of enforcement discretion on innovation in summing up the competing views expressed in reactions to its Draft Guidance on LDTs:

While excessive oversight can discourage innovation, inadequate and inconsistent oversight in which different test developers are treated differently can also discourage innovation by making it difficult for high-quality test developers to compete with poorer performing counterparts . . . When patients and providers discover that results they relied upon to make treatment and/or diagnostic decisions were inaccurate, their confidence in laboratory testing may be compromised . . . Appropriately tailored oversight can facilitate the development of analytically and clinically valid tests and the generation of the evidence health care providers and patients need to make well-informed decisions.[117]

Similar competing considerations inform FDA’s contemplation of how to regulate AI and machine learning systems. The agency will face one set of potential concerns if the technology is “locked” prior to marketing so that the algorithm will always provide the same result in response to the same input.[118] In such a case, the algorithm might not be tailored to provide accurate responses for critical subpopulations, or more generally might be poorly fitted to determine the true relationship between its inputs and its outputs.[119] But the agency would face different challenges if it permitted an algorithm to change as it learns from new data generated in the course of further experience.[120] This feature makes premarket regulation problematic as a mechanism for quality oversight. At what point in the lifecycle of a continuously changing algorithm is it time for further regulatory review?

FDA has proposed for discussion a “total product lifecycle” approach to regulation—which may require additional statutory authority—that relies heavily on manufacturer vigilance and best practices to provide reasonable assurance of safety and effectiveness of products that change over time.[121] FDA has worked with regulators in other countries under the auspices of the International Medical Device Regulators Forum (IMDRF) to develop this approach and to harmonize expectations for the regulatory treatment of these technologies in order to promote patient safety while fostering innovation.[122] As envisioned, regulators or third party evaluators would assess the culture of quality and organizational excellence of a particular company in a precertification program to ensure that manufacturers will monitor their devices to continually manage patient risks throughout the product lifecycle.[123] FDA would conduct premarket review for those devices that require it, establish clear expectations for manufacturers to continually manage patient risks throughout the product lifecycle, and require ongoing postmarket performance reporting and transparency.[124] Product changes that change the intended use would require a new premarket submission.[125] Although the plan has not yet been finalized, FDA reaffirmed its approach in January 2021.[126]

By reducing the regulatory burden on incremental product changes, this approach would permit software firms to perform their own quality oversight as they continually update their products. But the focus on company culture may offer less relief to new AI user innovators such as health systems and insurers that do not have the same history of exercising quality oversight over software products that seller innovators can demonstrate.

3. Implications for Medical AI

The regulatory implications for the technologies considered in this article are mixed. Some software functions—including “back office” administrative tasks such as billing and insurance reimbursement, general wellness and healthy lifestyle support, and electronic health records—may be categorically excluded from regulation as devices, although the boundaries of the excluded categories may be blurry enough to encourage prior consultation with FDA to be sure. Even for categorically excluded software functions, the Cures Act gives FDA authority to override the exclusion by finding that it is “reasonably likely to have serious adverse health consequences.”[127]

AI technologies that pertain more directly to diagnosis and treatment of patients will likely continue to meet the statutory definition of devices, although FDA might choose to regulate them with a light touch. The FDCA gives FDA considerable flexibility to classify medical devices into three different risk categories with different levels of regulatory controls.[128] FDA may also exercise enforcement discretion rather than exercising its full authority to regulate some devices that it believes pose low risk to the public.[129] For example, FDA recently stated in draft guidance that it does not intend to enforce compliance with the applicable device requirements of the FDCA for CDS functions intended to inform clinical management for non-serious situations or conditions, even when health care providers are unable to independently review the basis for the recommendation.[130] Discretionary forbearance from regulation under circumstances specified in nonbinding guidance documents may impress upon innovators a lurking possibility of regulation, allowing FDA to monitor new technologies informally without expending the administrative resources necessary for premarket approval or clearance of every product.

These provisions leave considerable uncertainty as to whether and how far FDA will assert regulatory authority, making it advisable for innovators to consult with FDA to avoid surprises. For some innovations, such as the Duke Sepsis Watch System, developers are doing just that.[131] When innovators work closely with regulatory officials as they design their products, FDA has an opportunity to oversee and guide product development, and perhaps to decide that the product requires a more robust process of premarket clearance or approval. (And, to the extent that companies’ involvement with FDA may lead the agency to establish regulatory barriers for other entrants to surmount, they may have anticompetitive effects as well.) Other innovators, however, have described little to no interaction with FDA officials.[132] These include health systems with actively running AI systems that make predictions and recommendations about patient care.[133]

In sum, the weight of FDA regulation of medical AI, as felt by innovators, appears to be fairly light—at least relative to the regulation of other medical devices. The landscape is complex, and developers of medical AI technologies intended for patient care face considerable uncertainty about whether and to what extent FDA will regulate these technologies as medical devices. The FDCA allows leeway for FDA to determine the appropriate level of regulatory controls for different devices depending on its assessment of the risks they present. FDA may nonetheless refrain from regulating some of these technologies, at least for now. In the face of uncertainty, some innovators may consult with FDA as they develop new technologies, allowing FDA to maintain oversight and to guide product development in ways that give it considerable control to mitigate patient risks.

B. Patent Law

Patents are typically considered an important incentive for biomedical innovation by product sellers. In theory, patent law provides a unitary system of legal rights for inventions in all fields of technology.[134] In practice, some industries rely on patents more heavily than others,[135] and courts and legislatures have adapted in a variety of ways.[136] The pharmaceutical industry consistently reports that patent incentives are essential to its willingness to invest in new drug development, while the story is more mixed in other fields.[137]

In medical AI, patents may be both less necessary and less powerful—and at least anecdotally, carry accordingly less weight in the innovation decisions of some user innovators.[138] They may be less necessary than in other biomedical innovation because AI systems and the data underlying them are often comparatively easy to protect through trade secrecy. In addition, to the extent that AI systems—especially those developed in-house—are truly bespoke innovations that respond to location-specific problems and would not fit other users, there may be little risk of free-riding by competitors, removing a classic justification for patents and making them less necessary.[139]

Patents are less powerful in this context for more complex reasons. Although many firms are pursuing patents on medical applications of AI and ML throughout the world,[140] the patent eligibility of these inventions under U.S. law is nonetheless in some doubt in light of recent case law.[141] Some scholars have also argued more broadly that aspects of patent doctrine (such as requirements for nonobviousness and disclosure) make it a poor fit for AI,[142] while others argue that the patent system can adapt as it has done before for other new technologies.[143] The U.S. Patent & Trademark Office (PTO) has recently reaffirmed its commitment to AI-related IP rights.[144] Nevertheless, in an environment of uncertainty, patents appear to provide weaker incentives for medical AI innovation.

1. Patent Eligibility

Four decisions from the U.S. Supreme Court between 2010 and 2014 revived and extended long dormant judicial limitations on patentable subject matter for “laws of nature, natural phenomena, and abstract ideas.”[145] Two decisions in particular have created uncertainty as to the patent eligibility of inventions in two fields that converge in medical AI: medical diagnostics and computer software.[146]

In Mayo v. Prometheus,[147] the unanimous Court relied on the “laws of nature” exclusion to invalidate a patent on a method of optimizing treatment with a drug by measuring drug metabolite levels in a patient’s serum and comparing them to specified reference values to determine the need to adjust the dosage.[148] The Court reasoned that the underlying relationship was a law of nature, and that the patent’s other claim elements (administering the drug and measuring metabolite levels in a patient) did not add enough to make the process patent-eligible.[149]

Two years later, in Alice v. CLS Bank,[150] the Court relied on the exclusion for “abstract ideas” to invalidate a patent on a computer-implemented method for mitigating settlement risk in a transaction.[151] The Court set forth a two-step patent eligibility test (the Alice/Mayo test): (1) is the claim directed to one of the judicial exclusions; and (2) if so, is there “an ‘inventive concept’—i.e., an element or combination of elements that is ‘sufficient to ensure that the patent in practice amounts to significantly more than a patent upon the [ineligible concept] itself?’”[152]

Lower courts applying this test have invalidated hundreds of previously issued patents in recent years,[153] often ruling on patent eligibility as a matter of law at the outset of litigation without developing an evidentiary record.[154] The Court of Appeals for the Federal Circuit (“Federal Circuit”) and the PTO have largely fallen in line behind the approach of the Supreme Court, although sometimes with explicit disagreement and lament.[155] The result has been considerable uncertainty as to what remains patent eligible in the fields of software, business methods, and medical diagnostics.[156] As many patents are struck down, patent lawyers and agents learn to draft around revitalized exclusions,[157] and the Federal Circuit and PTO consider arguments to narrow the exclusions.[158] In this environment it is difficult to assess with confidence the patent eligibility of medical AI inventions—but it is straightforward to conclude that patents on medical AI face real challenges in meeting this requirement.[159]

2. Patent Disclosure Requirements

Beyond the threshold issue of patent eligibility, medical AI innovators may have difficulty satisfying patent law requirements for an enabling disclosure of how to make and use the invention, a written description of the invention, and claims that particularly point out and distinctly claim the invention.[160] Adequately enabling an AI system should be possible, whether through a complex written document for transparent systems or potentially through depositing data and algorithms for opaque systems.[161] But because the scope of enablement limits the scope of allowable patent rights, the resulting patent might be quite narrow if it is difficult to generalize beyond the very specific example that has been disclosed.[162] Opaque algorithms present similar challenges for satisfying the requirements of written description and claim definiteness: even when it is possible to state in words what the algorithm is, it may be challenging to claim it in broad enough terms for the patent to have commercial value.[163]

Compliance with the disclosure requirements of patent law, even when possible, has another notable downside for medical AI innovators: disclosure of data and algorithms may destroy more effective protection for these innovations as trade secrets.[164] Firms can and do keep data and algorithms secret in a way that is more difficult in the context of prescription drugs or physical devices, which may be vulnerable to reverse-engineering once they are made available to the public. Trade secret law provides remedies for misappropriation of economically valuable secret information.[165] Access to data and control of large datasets has considerable competitive value for AI innovators, an advantage they might well hesitate to surrender in exchange for patent rights of uncertain validity and scope—particularly for user innovators who do not require patents to benefit from using the innovation themselves and do not plan to sell it to a broader market.

Taken together, these doctrinal challenges reduce the power of the patent system as an incentive for innovation in medical AI—at least, relative to that system’s power in other areas of biomedical innovation such as drugs and conventional medical devices.[166] Patents are more difficult to obtain, of more doubtful validity when granted, and more likely to cover relatively narrow inventions. We do not claim that patents are unavailable or that the incentives are negligible; indeed, commercial firms have filed many applications for patents on inventions involving medical AI. Our point is simply that patent protection is more uncertain for this technology, and thus that patents are less powerful shapers of innovation incentives. But the insecurity of patent protection is more likely to matter to seller innovators whose profit expectations depend on sales to a broader market of consumers than to user innovators who are adequately motivated to innovate by the expected benefits of using a customized innovation themselves.

C. Insurance Reimbursement

A third innovation policy lever that plays a different and diminished role in the AI space is insurance reimbursement. In most of the examples described in Part II, AI innovators will have difficulty obtaining direct reimbursement for the use of their AI technology from insurers, whether public (such as Medicare and Medicaid) or private.[167] To be clear, as noted in Part II, some AI tools may make health care delivery more efficient and enable providers to earn revenue on that basis, perhaps by collecting more insurance reimbursements for treating more patients. But the lack of direct reimbursement for the innovation itself distinguishes these AI innovations from traditional health care technologies like pharmaceuticals or medical devices. Manufacturers of typical new products can expect that their products will be reimbursed by insurers—and the prospect of reimbursement factors into innovation decisions.[168] Insurance will not provide reimbursement for the cost of using the innovations we consider, nor will reimbursement decisions provide quality oversight in this context.

Insurance reimbursement has not traditionally been recognized as part of the innovation policy toolkit, but in recent years scholars have increased their focus on insurance as a key driver of innovation incentives for health care technologies.[169] Insurance reimbursement functions very much like an innovation prize.[170] It is an ex post reward provided for the development of an innovative medical technology, funded largely by public subsidies, and reserves a relatively large role for the government or private insurers in setting the size of the award.[171] Insurers’ decisions to provide reimbursement for a new drug or device create a market for that product, and innovators respond by investing in the development of products that they expect will find a ready market (in the form of insurance reimbursement) once they make them available.[172]

The creation of Medicare Part D is an example of these dynamics. When Congress passed Medicare Part D in 2003,[173] it provided a prescription drug benefit to many Medicare enrollees who had previously lacked coverage entirely, or who had less comprehensive coverage.[174] As a result, Part D expanded the potential market for pharmaceutical companies by both increasing the number of seniors with the ability to pay for their products and by increasing the prices that pharmaceutical companies could expect to recoup on sales to seniors who previously had less remunerative insurance.[175] Economists studying Part D found that after its passage, pharmaceutical investment increased in drug classes with higher Medicare market share.[176] Economists analyzing other market-creating policy changes (such as particular coverage mandates) have found similar results.[177]

But direct insurance reimbursement is unlikely for these new AI technologies. Some functions are not directly reimbursable at all, such as AI systems that help schedule emergency rooms or reduce patient waiting time. Risk predictors that are routinely run for all patients, such as predictors of sepsis or readmission risk, are similarly unlikely to be directly billed or reimbursable. Instead, they are more likely to be folded into overall facility charges. Providers may nonetheless recover these costs diffusely, as they seek reimbursement for the care they provide as a whole.

To be sure, there may be financial incentives to use new health care technologies even without specific insurance reimbursement for them. AI technologies that have the potential to increase clinical volume (or even the explicit goal of doing so), such as the tool for reducing capacity strain from Cedars-Sinai discussed in Part II,[178] would likely enable providers to earn more money for the increased services they provide, even if they would not be able to bill directly for the use of the AI product. And some administrative AI tools can increase insurance reimbursement for non-AI services by scouring medical records for billable efforts or diagnoses that may not have been coded for reimbursement.[179]

Other forms of indirect financial incentives come from attempts to alter payment methodologies to reimburse providers for the value, rather than the volume, of the care they provide. Some are more formal: the Department of Health and Human Services (HHS) has been working to increase the amount of its reimbursement that is based on quality or value, both in the hospital setting and in the outpatient physician setting.[180] Prior to these and other reforms, providers may have obtained more reimbursement if a patient suffered an avoidable complication (such as a fall or certain hospital-acquired infections), as the providers could then bill for the treatment of that complication on top of their earlier services.[181] But if providers can no longer bill for treating complications like these—or if they even face financial penalties for their occurrence—health systems may have greater incentives to develop and adopt AI technologies that would reduce adverse events, such as the Sepsis Watch program,[182] or UT Southwestern’s readmission risk predictor.[183] Some changes are informal: using AI technology to improve efficiency or quality may provide leverage to negotiate higher reimbursement rates for care. At least one health system has been able to negotiate higher rates for office visits in part because its predictive algorithms decrease the number of hospitalizations.[184]

Although through these mechanisms the development of medical AI may increase overall reimbursements to user innovators, these indirect reimbursements are less amenable to manipulation by policymakers than direct reimbursements. If policymakers want greater innovation in a certain type of vaccine, for instance, they can precommit to reimburse that vaccine at a higher rate to drive that innovation; this type of explicit, reimbursement-driven incentive structure, and its associated policy tools, is less prominent for medical AI.

Another innovation-related benefit of insurance is also difficult to deploy in this space: the role insurance plays in ensuring quality oversight of new health products. Insurance companies often serve an important function as independent evaluators of new medical technologies, demanding information on the reasonable necessity of particular products before agreeing to provide reimbursement for them. Even if laws require insurance coverage, as they do for many payers in the case of new drugs,[185] insurers may use information about drug effects to create preferred drug lists, favoring some drugs over others, or to create prior authorization requirements that impose additional administrative hurdles on physicians and patients before they can obtain a particular product.[186]

Insurers’ quality oversight role is particularly important when FDA does not require data from high quality studies as a condition of market access.[187] Although FDA review of new medical devices is far less stringent than review of new drugs, insurers often demand that additional criteria are met—beyond FDA authorization—before they will agree to cover a particular product.[188] Insurer review may thus fill gaps in FDA’s information-forcing function for less regulated products.[189] But when FDA oversight is weak or unlikely, and insurers are not directly reviewing AI tools for possible reimbursement, there may be little to no external oversight of the quality of these AI innovations.

In some cases, it will be readily apparent to the innovator even without such oversight that an AI tool is not effective for its intended use. If sepsis rates do not drop, or if thirty-day readmission rates are unchanged, a health system may re-evaluate its approach for that AI product. But in other cases, it may be more difficult to identify whether an AI tool is safe and effective. Perhaps the tool does work to drive down the rate of the complication targeted by the algorithm—but with many false positives, leading many patients to receive unnecessary treatment as a result. Or perhaps the tool benefits some racial groups and not others.[190] A lack of direct insurance reimbursement makes it much more difficult for insurers to meet these needs for additional quality oversight.

III. Implications

The previous two parts have catalogued how user innovators face substantially different incentives and development costs in the medical AI context than seller innovators face for more familiar biomedical products like traditional medical devices or drugs. Patents and insurance reimbursement provide weaker incentives for these innovations. But development costs and regulatory barriers to entry are lower as well. Many of these medical AI innovations have been developed and used without going through a potentially lengthy FDA approval or clearance process. Nor do they require building new production facilities. Although assembling data and informatics capacity can be expensive for some institutions, those that have these resources at hand may develop and implement AI products relatively inexpensively. Lower development costs make innovation a reasonable investment even when expected benefits are also lower, and even when benefits come primarily from internal efficiency improvements rather than from external market sales.[191]

This legal landscape may be especially advantageous for user innovation by health systems and insurers. We consider three interconnected implications for innovation processes. First, the availability and quality of data impacts both who can innovate and the quality of their innovations. Second, the products that result are often custom solutions to local problems. Policy makers may have fewer tools for influencing these innovations than they have for seller innovations designed for broader markets, with possible implications for social welfare. Third, and finally, these products face less oversight than do many biomedical products, leading to the risk of quality problems that may be difficult to detect.

A. Data Control

A substantial barrier to successful innovation in medical AI seems to be the availability of high-quality data. AI requires large amounts of data, and assembling, formatting, and curating data from multiple and heterogeneous sources is an expensive task.[192] This gives an advantage to entities that already possess substantial stores of data—among them large health systems (especially academic medical centers) and health insurers.[193] Indeed, even those data-holders uninterested in developing their own projects have gotten into the game. The Veterans Administration (VA), while engaging in minimal internal AI projects for reasons including a lack of expert programmers, has entertained many requests for collaboration and tool-building, largely based on the value of its substantial longitudinal dataset.[194] And insurers have used the lure of their data as a prize for developers who create useful algorithms.[195] Although advantageous for large health systems and insurers, the central role of data access can have problematic implications.

For one thing, if in-house development relies on in-house data, only some entities will have access to the very large datasets necessary for training high-quality AI. They may be able to create solid products for their own use (though the performance of single-system development even with very large datasets may lag the performance that could be obtained from training similar algorithms across multi-system data.)[196] Entities with smaller datasets are more likely to introduce quality problems or biases into resulting AI systems. A particularly dangerous middle ground includes those entities that have sufficient data to plausibly create useful AI products, but insufficient data to weed out possible bias or performance issues. Moreover, the ability of some powerful actors within the health system to develop in-house products based solely on their own data may reduce the impetus for broader data-sharing efforts, including the creation of large-scale centralized datasets that multiple institutions could use.[197] The competitive value of data as a resource for AI product development may further encourage the trend of data-hoarding, slowing the development of better tools trained on larger aggregations of shared data.

To be sure, some problems may truly demand localized solutions, as the user innovation literature points out; perhaps the factors that drive C. diff. infection in one hospital are fundamentally different from those that drive infection in another.[198] But perhaps not. At a minimum, it would be worth probing why such a biological result seems so context-dependent—a project that would surely benefit from interrogating a broader dataset assembled from multiple institutions.[199] For other products, like a staffing prediction or patient-flow model, context specificity seems likely to be typical. It might nonetheless seem problematic that smaller entities may have smaller datasets for their own context and are therefore at higher risk of developing erroneous AI systems.

If the availability of data is a major hurdle, we should also expect to see substantial discrepancies among health entities in their ability to develop their own AI systems at all. Small community health systems, for instance, are much less likely to have the data capacity (or, for that matter, the information-technology capacity) to develop their own algorithms. To the extent that self-developed AI systems adapted to a particular system become important for providing high-quality health care, for interacting with insurers or other systems, or for maintaining a competitive level of efficiency, smaller systems will be especially disadvantaged. If smaller systems react by attempting to adopt AI products developed in larger systems, systematic quality or bias problems may occur as a result of differences in patient populations and care patterns, as described in Section IV.C below.[200]

On the other hand, the user innovation literature teaches that in many contexts innovations initially developed by lead users at the forefront of their field to meet their own needs can provide prototypes that point seller innovators towards opportunities to create more standardized versions to supply a larger market of users.[201] When large academic medical centers innovate successfully, smaller providers will not necessarily try to rely on their own much smaller datasets to compete, but may instead look to seller innovators to provide better commercial AI products. We see evidence that commercial sellers are already filling that role for some of the innovations we consider here, including the commercial firm Pieces, which developed the “Pieces Predict” commercial product based on a thirty-day readmission model developed at UT Southwestern.[202] Other institutions with large datasets, such as the VA, are collaborating with commercial developers from the start to use their data to develop new AI products rather than hoarding their data exclusively for user innovation.[203]

Policy makers could further support the development or licensing of high-quality AI tools for smaller health systems or community settings in a number of ways to ensure equitable access to quality care. CMS, with its increasing focus on rural health care,[204] might work with NIH to establish prize funds for the development of AI tools applicable to these settings. These agencies might consider whether data from multiple similarly situated facilities could be pooled to accomplish these goals, with pooling decisions dependent on metrics of interest for the relevant AI tool. Some, including one of us, have called for broad disclosure of privately held data and algorithmic development information with the goal of creating public goods for the sake of wider innovation.[205] Notably, such disclosure mandates would be in some tension with the idea of lead users developing their own proprietary, cutting-edge solutions for the purpose of competitive advantage. On the commercial side, regulators might support efforts encouraging larger systems that develop successful products for their own use to license them to commercial firms to test and refine them further for more widespread use by smaller systems. To the extent that the federal government itself is the repository for a significant amount of this data (including through the VA), federal policymakers might be encouraged to play this role themselves.

B. Customized Products

As user innovators, health systems and insurers are likely to develop different products for different needs than firms developing biomedical innovations for sale to others. Most obviously, as the user innovation literature suggests, user innovators (whether health systems, insurers, or some other type of entity) will tend to develop products that are closely suited to their own particular needs.[206]

AI may be especially useful for such customized solutions. Where data formats, care patterns, and health problems are specific to a particular context, as the fragmented nature of U.S. health care suggests will be common,[207] AI products developed by users may be an effective solution that would be foreclosed if development costs and legal hurdles were higher. Thus, health systems can develop products to model and predict patient flow that are particularly responsive to their own patient dynamics.[208] Duke’s Sepsis Watch AI tool illustrates the importance of user-developed contextual knowledge to technical AI success. This app, developed by a team at Duke University, uses machine learning to determine a patient’s risk of developing sepsis; however, in order for Sepsis Watch to work, successful implementation has required disruption to the ordinary workflow in the emergency department; Duke ER nurses, assigned as primary users of the app, needed to reverse the typical ER chain of command to alert ER physicians of the app’s findings. This kind of disruption may have doomed another AI product, but Duke’s Sepsis Watch tool has been very successful at significantly reducing sepsis-induced patient deaths at Duke Health. This success is at least partially due to the fact that Sepsis Watch was created for the specific context of the Duke emergency department. Madeleine Eilish, a member of the Duke team evaluating the implementation of Sepsis Watch at Duke, cited the context-specific user development of Sepsis Watch as one of the keys to the app’s success, noting that “the tool was adapted for a hyper-local, hyper-specific context: it was developed for the emergency department at Duke Health and nowhere else.”[209]

Some might question whether it makes sense as a matter of regulatory policy to encourage such narrowly targeted innovative efforts.[210] Innovators engage in efforts based on the data that they have, the incentives they face in their own contexts, and the particular problems that seem most pressing to them. Sometimes this can reflect useful customization to genuinely local problems, as the user innovation literature suggests.[211] Where local innovation is siloed both by the presence of the data and the contextually local nature of the problem, it may decrease the opportunities for learning from others—a problem that may be especially aggravated if privacy laws make it more difficult to share data with other institutions.[212] Although some user innovators freely reveal their innovations, they may have little incentive to invest significant resources in diffusion of their innovations to others.[213] On the other hand, academic medical centers may be more likely than other user innovators both to publish their innovations and to license them to commercial firms.

Often the lead users who innovate to address their own idiosyncratic needs are harbingers of future trends.[214] It may be difficult to tell at the point of innovation whether an academic medical center is addressing a purely local problem, or whether they are ahead of the field in recognizing and addressing a problem that others will also recognize and want to address down the road. Von Hippel and Torrance use the term “innovation wetlands” to highlight both the value of user innovation and its vulnerability to policy choices focused on short-term assessments.[215]

Where context is not particularly important, the advantage of user innovation over seller innovation may be smaller, although lead users may still have an important role to play in pointing out emerging needs. Problems that are especially pressing for one academic medical center may reflect systematic problems that are best tackled at a broader level, rather than having individual institutions each devoting limited innovation resources to addressing the same problems locally. For instance, although there is some user innovation related to billing,[216] examples are not plentiful. On the other hand, there are already several commercial products available that use AI for billing optimization, suggesting that context specificity may be less important for this type of product and it can more readily be developed by outsiders for sale to a broader market. The commercially developed 3M 360 Encompass System,[217] for example, scans medical records to identify services that were provided to a patient, suggesting which codes can be billed to insurers.[218] On the insurer side, Kirontech touts the use of unsupervised machine learning and natural language processing to validate claims and to detect fraudulent health insurance claims in real time.[219]

Since commercial incentives are greater for seller innovators to address less contextual problems, policymakers might consider how best to help user innovators address more contextual problems—though as we have highlighted, much such innovation is already taking place. Policy support for user innovation might focus on grants for data infrastructure, best practices for AI development and validation, or training data scientists to create contextual AI solutions.[220]

C. Decreased Quality Oversight

Medical AI innovations are likely to be implemented with less independent quality oversight relative to other medical devices, including commercially developed point-of-care AI. As described above in Sections III.A and III.C, quality oversight for some new health technologies comes from FDA regulation and insurance coverage determinations, both of which provide some assessment of quality for new products. These two players have a smaller role in assessing many of the products we describe here, leading to less oversight for safety and effectiveness. Thus, many algorithms that may have an impact on patient care receive less external vetting before they are implemented, and may thus cause harm because of undiscovered error.[221]

We need not look far to see how such error might occur.[222] One team at Mount Sinai developed an AI algorithm to identify pneumonia based on patient chest x-ray images.[223] The algorithm was trained on Mount Sinai data, where many chest x-rays came from patients with pneumonia, and detected pneumonia 93% of the time.[224] When the algorithm was tested on images from the NIH and the Indiana Network for Patient care, performance dropped sharply. One reason was that the rates of pneumonia at those sites were lower, and the patient populations were different. Another reason was that the algorithm had learned to predict outcomes based on whether the image came from a portable x-ray machine or a fixed machine—and the latter is typically used on sicker patients.[225]

Had this algorithm been developed and deployed for clinical care, such a performance pattern would be highly problematic. The algorithm might appear to function decently at Mount Sinai, since it would be relying on the same provider-created information (e.g., the use of portable x-rays) that it was trained on—but it would not provide as much useful information as the hospital thought, and its performance would degrade over time as care patterns shifted. If, for instance, the hospital shifted to using portable x-ray machines more broadly, the algorithm, relying on an unreliable proxy signal, would suggest that many more people had pneumonia. An even worse situation would arise if the algorithm were deployed at other sites, such as the Indiana network, without further testing; performance would be substantially worse, but provider-users could easily be none the wiser. In this case, testing by the developers to evaluate performance in different contexts provided the type of quality oversight we might hope for, perhaps demonstrating independent evaluation of the sort that FDA is emphasizing in its recent thinking about regulating medical AI.[226] Unfortunately, the vast majority of AI image analysis algorithms are tested at only one hospital, making it all too likely that innovators will fail to notice such problematic development patterns on their own.[227]

The failure of some AI systems will be obvious to users—readmissions might increase rather than decrease—but others may go undetected. Health outcomes are notoriously difficult to attribute to a particular intervention, which is why health technology is described as a credence good and why formal regulatory oversight is justified.[228] For other problematic outcomes, the results may be observable but not actually observed; if hospital algorithms result in patients suffering ills that they self-treat with over-the-counter medications (being discharged too early, for instance), those problems might never come to the attention of the innovator.[229]

Most perniciously, biased results—prioritizing wealthy or white patients, for instance—might well accord with a hospital’s bottom-line incentives of efficiency and revenue maximization, even if they are socially and morally repugnant and might otherwise be detected and blocked by a regulator. This sort of problem, too, has already occurred on a substantial scale.

As Ziad Obermeyer and colleagues documented, Optum, a unit of the large insurer UnitedHealth Group, developed an AI system in-house to predict which patients were likely to have medical complications, with the aim of providing outpatient guidance services to particularly high-risk patients.[230] This system was developed, validated, and deployed in due course, and influenced the care received by many patients. But, as Obermeyer’s team found, the algorithm was not at all neutral. The developers used a reasonable proxy for complexity of patient care: how much was spent on the patient’s care. But they did not account for the fact that Black patients and other minority patients receive substantially less care, and consequently cost less, for reasons of systemic bias or resource constraints rather than because of medical differences. Accordingly, the algorithm predicted white patients were at substantially higher risk than Black patients with the same ailments, with the result that white patients received more outpatient coordination services. The bias went unnoticed by the developers, presumably because the outcomes aligned reasonably with the algorithm’s goals of reducing costly complications, as measured by later costs. Eventually, oversight by academic researchers caught the problem, at least in this instance, and Optum agreed to work with Obermeyer’s team to fix the problem going forward.[231] But this sort of bias could readily arise in unregulated use of medical AI.

We do not mean to overstate the point about lessened oversight from key players. In particular, FDA retains the authority to regulate software as a medical device, even if the software initially falls into one of the Cures Act’s exclusions.[232] But policymakers should consider what role other players might have here as well. Learned societies could play some part in determining the quality of medical AI within their respective bailiwicks, offering views as to when these products should and should not be used. The American College of Radiologists, for instance, could evaluate medical AI that evaluates radiological images, though homegrown products might fly under the radar of learned societies as well.[233] Another possibility would be to establish a system of oversight with a lighter touch, more analogous to the one used by the Centers for Medicare and Medicaid Services in their administration of the Clinical Laboratory Improvement Amendments program (CLIA).[234] Although FDA may be exercising its enforcement discretion to relieve LDTs of the full burden of device regulation under the FDCA, those laboratories still receive a form of regulation through CLIA, and laboratories must obtain CLIA certification if they wish to receive reimbursement from Medicare and Medicaid more generally.[235] Indeed, FDA and other regulators may ultimately design such a mechanism in the course of ongoing efforts to adapt device regulation to software as a medical device.[236] Hospital accreditation mechanisms are another potential source of oversight, with accreditation being required by states and the federal government as a condition of continued operation and reimbursement.


The development of AI tools and the widespread adoption of electronic health records have fostered user innovation in medical AI in a legal landscape that looks different from that applied to more familiar forms of biomedical innovation to develop new drugs and traditional devices. Patents are less certain and less powerful; FDA remains important but lurks in the background rather than looming over product entry, and insurance reimbursement provides fewer direct incentives and less quality oversight. And yet these products offer great prospects for improving patient care, whether they are predicting risky outcomes, monitoring vital signs, managing patient flow, or allocating resources. There is a world of medical AI innovation occurring inside health systems and insurers that differs from commercial product development in important ways. We do not suggest that we have fully characterized this world, or that we know how to solve the problems that may arise. Policy interventions in this space are choices, and there is likely no optimum, but it is important to be alert to the innovation landscape while making policy choices. Although policy interventions that are powerful with respect to seller innovation, whether patents, regulation, or reimbursement, are less powerful for user innovation, that does not imply that the incentives need to be changed to mimic their effects elsewhere; other tools may be better suited to shape the challenges we observe, or to nurture the positive trends. Nor is the world of user innovation in this space fully separate from the world of seller innovation; products that start their life as user innovation in a single health system may well be spun out into startup companies or acquired by existing manufacturers, making the transition from user innovation to a more classic commercialization model, where different legal regimes and incentives apply. Nevertheless, the space of user innovation in medical AI is worth further examination. We have aimed here to point out its major features, to indicate its importance, and to raise the call for future study.

  1. WNP is Professor of Law, University of Michigan Law School; Core Partner, Centre for Advanced Studies in Biomedical Innovation Law at the University of Copenhagen; and Co-PI, Project on Precision Medicine, AI, and the Law at the Petrie-Flom Center at Harvard Law School. RES is Treiman Professor of Law, Washington University in St. Louis School of Law. RSE is Robert and Barbara Luciano Professor of Law, University of Michigan Law School. For detailed and thoughtful comments on earlier drafts, we thank Chris Buccafusco, Glenn Cohen, Kevin Collins, Nathan Cortez, Dmitry Karshtedt, Erika Lietzan, Sarah Rajec, Ana Santos Rutschman, and Joy Xiang. For helpful comments and conversations, we thank Stefan Bechtold, Ed Fox, Rich Friedman, Brett Frischmann, Gabriel Rauterberg, Leora Horwitz, Mauritz Kop, Nina Mendelson, Lisa Larrimore Ouellette, Jason Rantanen, Michael Risch, Margo Schlanger, Rebecca Scott, Mark Sendak, Karandeep Singh, and Jinfeng Su. This project benefited from comments at the Michigan Law School Fawley Workshop; the Northwestern Pritzker School of Law Faculty Workshop; the Intellectual Property Scholars Conference at Stanford Law School; the Machine Learning in Health Care Conference at Duke; the joint ETH Zurich, University of St. Gallen & University of Zurich Lecture Series on the Law and Economics of Innovation; and the Junior IP Scholars Association Virtual Workshop. Many thanks to Maydha Vinson for outstanding research assistance. The work of WNP and RSE was supported by the Cook Fund at the University of Michigan Law School, and WNP received further support from the Novo Nordisk Foundation (NNF17SA0027784). All errors are our own.

  2. . AI Central, Am. Coll. Radiology Data Sci. Inst., []; Andrew A.S. Soltan et al., Rapid Triage for COVID-19 Using Routine Clinical Data for Patients Attending Hospital: Development and Prospective Validation of an Artificial Intelligence Screening Test, 3 Lancet Digit. Health E78 (Dec. 11, 2020), [].

  3. . See, e.g., W. Nicholson Price II, Regulating Black-Box Medicine, 116 Mich. L. Rev. 421 (2017).

  4. . See W. Nicholson Price II, Artificial Intelligence in the Medical System: Four Roles for Potential Transformation, 21 Yale J.L. & Tech. (Special Issue) 122 (2019).

  5. . To be clear, other laws do shape innovation in this space, including privacy laws, human subjects research protections, and trade secrecy, though they are not our focus here. See, e.g., Sharona Hoffman, Electronic Health Records and Medical Big Data: Law and Policy (2016) (providing an overview of medical big data law); W. Nicholson Price II & I. Glenn Cohen, Privacy in the Age of Medical Big Data, 25 Nature Med. 37 (2019) (describing medical privacy laws); Kayte Spector-Bagdady & Reshma Jagsi, Big Data, Ethics, and Regulations: Implications for Consent in the Learning Health System, 45 Med. Physics e845 (2018) (describing human subjects protection law); Arti K. Rai, Risk Regulation and Innovation: The Case of Rights-Encumbered Biomedical Data Silos, 92 Notre Dame L. Rev. 1641 (2017) (describing regulation and secrecy).

  6. . We recognize that “difference” demands a baseline. We focus on the biomedical innovation baseline because the actors we consider here operate largely in the world of drugs, devices, and other biomedical innovations. But we recognize that this is not the only potential baseline. Interesting insights could come from focusing on the different baseline of software innovation generally, and considering how medical AI differs from other software, where patents are of disputed value and FDA regulation and insurance reimbursement are non-players. See generally, e.g., Julie E. Cohen & Mark A. Lemley, Patent Scope and Innovation in the Software Industry, 89 Cal. L. Rev. 1 (2001) (analyzing the scope of protection that should be afforded to software patents); John R. Allison & Ronald J. Mann, The Disputed Quality of Software Patents, 85 Wash. U. L. Rev. 297 (2007) (considering the quality of software patents); Colleen V. Chien, Reforming Software Patents, 50 Hous. L. Rev. 325 (2012) (placing reform proposals for software into historical context). Such an analysis could examine the impact on software development of heightened regulatory scrutiny relative to an all-software baseline, rather than the diminished scrutiny relative to medical devices generally that we discuss here. Although that is not this paper, our analysis does explore the ways in which software-like features of medical AI pose challenges for both FDA regulation and patent protection of these medical innovations. See infra Sections III.A, III.B.

  7. . See Eric von Hippel, Sources of Innovation (1988) [hereinafter von Hippel, Sources]; Eric von Hippel, Democratizing Innovation (2005) [hereinafter von Hippel, Democratizing]; Eric von Hippel, Free Innovation (2017) [hereinafter von Hippel, Free].

  8. . Von Hippel’s earlier work contrasts user innovators with “manufacturers,” but we follow Katherine Strandburg’s usage in substituting the term “seller innovators” for the reasons she explains: “not all seller innovators are manufacturers, while manufacturers can be user innovators of industrial processes and manufacturing equipment.” Katherine J. Strandburg, Users, Patents, and Innovation Policy, in The Oxford Handbook of Intellectual Property Law 725 n.1 (Rochelle C. Dreyfuss & Justine Pila eds., 2018).

  9. . Id. at 63–76.

  10. . See, e.g., Rebecca S. Eisenberg, The Role of the FDA in Innovation Policy, 13 Mich. Telecomm. & Tech. L. Rev. 345, 370 (2007); Amy Kapczynski & Talha Syed, The Continuum of Excludability and the Limits of Patents, 122 Yale L.J. 1900 (2013); Rachel E. Sachs, Prizing Insurance: Prescription Drug Insurance as Innovation Incentive, 30 Harv. J.L. & Tech. 153, 193, 201–08 (2016) [hereinafter Sachs, Prizing Insurance].

  11. . von Hippel, Democratizing, supra note 7, at 33–44.

  12. . See, e.g., Barbara J. Evans & Frank Pasquale, Product Liability Suits for FDA-Regulated AI/ML Software, in Innovation and Protection: The Future of FDA Medical Device Regulation (I. Glenn Cohen, Timo Minssen, W. Nicholson Price II, Christopher Robertson & Carmel Shachar eds., forthcoming 2022); Price, supra note 3; Charlotte Tschider, Medical Device Artificial Intelligence: The New Tort Frontier, 46 B.Y.U. L. Rev. 1551 (2021).

  13. . See Letter from Angela C. Krueger, Acting Deputy Dir., Eng’g & Sci. Rev., Off. of Device Evaluation, to Janice Hagan, Regul. Couns., Hogan Lovells US LLP (Apr. 11, 2018), [] (classifying IDx-DR as a Class II medical device); Press Release, U.S. Food & Drug Admin., FDA Permits Marketing of Artificial Intelligence-Based Device to Detect Certain Diabetes-Related Eye Problems (Apr. 11, 2018), []; Aimee Breaux, With New CEO, Artificial Intelligence Start-Up Expects 2020 to Be ‘Year of Mass Adoption,’ Iowa City Press Citizen (Dec. 11, 2019, 4:42 PM), [] (“Abramoff said IDx has sold these devices to around two dozen hospital systems. . . In November, the company announced a new type of client, retail clinics. The company closed a deal with grocery store chain Albertsons to outfit five of the company’s retail clinics with IDx cameras.”).

  14. . As of September 2020, sixty-four machine learning or AI-based algorithms and devices had been approved by FDA. See Stan Benjamens, Pranavsingh Dhunnoo & Bertalan Meskó, The State of Artificial Intelligence-Based FDA-Approved Medical Devices and Algorithms: An Online Database, 3 NPJ Digit. Med. 1, 1 (2020). To be sure, there is substantial contestable space about what in this field counts as a “medical device;” we consider those questions to some extent in Section III.A.

  15. . We do not mean to suggest that commercial products are not available to serve these goals. For instance, Epic, the largest EHR vendor in the United States has developed and implemented a sepsis predictor, in addition to the health system-developed ones we describe in this Part. Meg Bryant, Epic, Cerner Control 85% of Large Hospital EHR Space, KLAS Reports, Healthcare Dive (May 2, 2019), []; Bill Siwicki, Health System Uses Epic EHR Communications Tech to Reduce Sepsis Mortality Rate by 20%, HEALTHCARE IT NEWS (Oct. 1, 2019, 1:02 PM), []. But even with the existence of tools like these, many health systems have chosen to develop their own products.

  16. . A previous line of demarcation between these entities had been the difference between products, regulated by FDA, and practice, regulated by the states and medical bodies. But the increase in tools like the ones discussed here, particularly in the clinical decision support area, have blurred or even eliminated this distinction.

  17. . Although the precise cost to develop a new drug is hotly debated, there is no question that pharmaceuticals are among the most costly new products to bring to market, with estimates typically placing the cost to develop a new drug at well over a billion dollars. See, e.g., Joseph A. DiMasi, Henry G. Grabowski & Ronald W. Hansen, Innovation in the Pharmaceutical Industry: New Estimates of R&D Costs, 47 J. Health Econ. 20, 20 (2016) (estimating pre-approval costs to be $2.558 billion); Jorge Mestre-Ferrandiz, Jon Sussex & Adrian Towse, Off. Health Econ, The R&D Cost of a New Medicine (2012), [] (estimating costs at $1.5 billion); Cynthia M. Ho, Drugged Out: How Cognitive Bias Hurts Drug Innovation, 51 San Diego L. Rev. 419, 426, 448–57 (2014).

  18. . Josh Makower, Aabed Meer & Lyn Denend, FDA Impact on U.S. Medical Innovation 28 (2010), [] (estimating the cost of developing a Class III medical device at $94 million).

  19. . By health system, we mean a set of health care organizations that are contractually affiliated with each other, particularly including the relationship between hospitals and outpatient physician organizations. See, e.g., Agency for Healthcare Rsch. & Quality, Defining Health Systems (Sept. 2017), []. While we recognize that individual hospitals may have slightly different incentives and capacities than health systems, for the sake of convenience we generally include individual hospitals within the broad term “health system.”

  20. . Of course, considerable research occurs in hospitals, particularly in academic medical centers, including developing new medical procedures or protocols. But legal scholarship in intellectual property and innovation policy has largely focused on the commercial development of new health care technology products, perhaps in part because the Patent Act explicitly bars the enforcement of patents granted on medical procedures. See 35 U.S.C. § 287(c); Jonas Anderson, Nonexcludable Surgical Method Patents, 61 Wm. & Mary L. Rev. 637, 657 (2020).

  21. . See, e.g., Thomas Bodenheimer, Uneasy Alliance: Clinical Investigators and the Pharmaceutical Industry, 342 New Eng. J. Med. 1539, 1539 (2000); Hamilton Moses III, Eugene Braunwald, Joseph B. Martin & Samuel O. Thier, Collaborating with Industry—Choices for the Academic Medical Center, 347 New Eng. J. Med. 1371, 1371–72 (2002).

  22. . Other scholars have applied the user innovation paradigm to a wide range of innovation settings. See, e.g., Katherine J. Strandburg, Users as Innovators: Implications for Patent Doctrine, 79 U. Colo. L. Rev. 467, 472 (2008) (biomedical research tools); Christopher Buccafusco, Disability and Design, 95 N.Y.U. L. Rev. 952, 1007 (2020) (products for people with disabilities); see generally, e.g., William W. Fisher III, The Implications for Law of User Innovation, 94 Minn. L. Rev. 1417 (2012) (other consumer goods).

  23. . See von Hippel, Democratizing, supra note 7, at 33.

  24. . See infra text accompanying note 180. The federal government also operates searchable databases, including Hospital Compare and Physician Compare, that enable searches by certain quality of care metrics, such as mortality rates. These databases may exert indirect pressures on those metrics as well. See Nathan Cortez, Regulation by Database, 89 U. Colo. L. Rev. 1, 8, 24, 56 (2018).

  25. . For an analysis of the translational process and a list of products in development, including many originating in academia, see Mark P. Sendak et al., A Path for Translation of Machine Learning Products into Healthcare Delivery, EMJ Innovations 19-00172 (Jan. 27, 2020), [].

  26. . Opacity to observers sometimes gives a comparative advantage to user innovators over seller innovators in other contexts as well. Von Hippel uses the term “sticky information” to describe information held by users that they cannot readily transfer to others. See von Hippel, Democratizing, supra note 7, at 67–70.

  27. . Jeeheh Oh et al., A Generalizable, Data-Driven Approach to Predict Daily Risk of Clostridium Difficile Infection at Two Large Academic Health Centers, 39 Infection Control & Hosp. Epidemiology 425 (2018).

  28. . Ctrs. for Disease Control & Prevention, Antibiotic Resistance Threats in the United States vii (2019), []

  29. . Benjamin Y. Li et al., Using Machine Learning and the Electronic Health Record to Predict Complicated Clostridium Difficile Infection, 6 Open F. Infectious Disease ofz186 (2019).

  30. . Id.

  31. . See, e.g., Xi Na et al., A Multi-Center Prospective Derivation and Validation of a Clinical Prediction Tool for Severe Clostridium Difficile Infection, 10 Plos One e0123405 (2015).

  32. . Oh et al., supra note 27.

  33. . Mark Sendak et al., Real-World Integration of a Sepsis Deep Learning Technology into Routine Clinical Care: Implementation Study, 8 JMIR Med. Informatics 1, 6 (2020).

  34. . Mark Sendak et al., “The Human Body is a Black Box”: Supporting Clinical Decision-Making With Deep Learning, FAT* 2020: Proceedings of the 2020 Conference on Fairness, Accountability & Transparency 99 (2020), [] (describing the development of Sepsis Watch as a sociotechnical system); Andrew Ng, The Batch (Apr. 15, 2020), [] (noting the deployment of Sepsis Watch to two community hospitals).

  35. . Sendak et al., supra note 33, at 8. In order to increase provider buy-in, “[t]hroughout the design, development, and implementation process, Sepsis Watch was described as a ‘tool’ to support physicians and nurses in the ED and the term ‘artificial intelligence’ was not used in any communication or presentation.” Id. at 10.

  36. . Press Release, Duke Translation & Commercialization, “Deep Sepsis” Licensed to Cohere Med (July 3, 2019), [].

  37. . See, e.g., Cortez, supra note 24, at 57.

  38. . Ruben Amarasingham et al., Electronic Medical Record-Based Multicondition Models to Predict the Risk of 30 Day Readmission or Death Among Adult Medicine Patients: Validation and Comparison to Existing Models, 15 BMC Med. Informatics Decision Makers 39 (2015); Sendak et al., supra note 25.

  39. . Our Products, Pieces Techs., [].

  40. . Jessica Kent, Applying Artificial Intelligence to Chronic Disease Management, Health IT Analytics (Sept. 18, 2020), [].

  41. . Hospitals have less incentive to reduce their costs of care if doing so will lower their overall reimbursement totals or profit margins. As such, these tools are likely to be more powerful where hospitals are operating in a managed-care or otherwise value-based context, rather than in a pure fee-for-service model. Russell Korobkin, The Efficiency of Managed Care “Patient Protection” Laws: Incomplete Contracts, Bounded Rationality, and Market Failure, 85 Cornell L. Rev. 1, 10–13 (1999).

  42. . Slideshow at 19, Michael Thompson, New Ways to Improve Hospital Flow with Predictive Analytics, Health Catalyst (Mar. 18, 2020), (slides on file with author).

  43. . Alexander Fenn et al., Development and Validation of Machine Learning Models to Predict Admission from Emergency Department to Inpatient and Intensive Care Units, 78 Annals Emergency Med. 290 (2021).

  44. . Benjamin A. Goldstein et al., Development and Performance of a Clinical Decision Support Tool to Inform Resource Utilization for Elective Operations, 3 JAMA Network Open e2023547 (2020).

  45. . Development and Implementation of a Hospital at Home Program in Wake County, Duke Inst. for Health Innovation (2021), []; Press Release, Ctr. for Medicare & Medicaid Servs., CMS Announces Comprehensive Strategy to Enhance Hospital Capacity Amid COVID-19 Surge (Nov. 25, 2020), [].

  46. . To be sure, many businesses in other industries must also manage their costs for these types of administrative burdens. But as noted above, these pressures loom large in the highly-fragmented U.S. health care system.

  47. . See, e.g., David U. Himmelstein, Terry Campbell & Steffie Woolhandler, Health Care Administrative Costs in the United States and Canada, 2017, 172 Annals of Internal Med. 1 (2020); Steffie Woolhandler, Terry Campbell & David U. Himmelstein, Costs of Health Care Administration in the United States and Canada, 349 New Eng. J. Med. 768 (2003); Karen Davis et al., Slowing the Growth of U.S. Health Care Expenditures: What Are the Options?, Commonwealth Fund 4 (Jan. 2007).

  48. . Telephone Interview with Anonymous Head of an Academic Medical Center’s Machine Learning Program (Dec. 30, 2020).

  49. . Id.

  50. . See, e.g., Sarah Kliff, Hospitals Knew How to Make Money. Then Coronavirus Happened., N.Y. Times (May 20, 2020), [].

  51. . Of course, non-academic medical centers are also made up of health care providers who may be individually motivated by genuine scientific and medical curiosity.

  52. . See, e.g., Nat’l Inst. Biomedical Imaging & Bioengineering, Research Funding: Artificial Intelligence, Machine Learning, and Deep Learning (2020), []; Amarnath R. Annapureddy et al., The National Institutes of Health Funding for Clinical Research Applying Machine Learning Techniques in 2017, 3 NPJ Digit. Med. 1 (2020).

  53. . Those financial implications can extend as well to individual academic innovators, who may sometimes found commercial spin-off companies from their academic work. See, e.g., supra note 39 and accompanying text.

  54. . Basic research describes research aimed at fundamental understanding more than applied uses for that understanding. See W. Nicholson Price II, Grants, 34 Berk. Tech. L.J. 1, 11–12 (2019) (describing funding for basic research).

  55. . See, e.g., Xi Luo, National Insts. Health Report, Large-Scale Network Modeling for Brain Dynamics: Statistical Learning and Optimization (2019),‌project_info_‌description.cfm?aid=9899033&icde []; Li Shen, Nat’l Insts. Health Report, Integrative Bioinformatics Approaches to Human Brain Genomics and Connectomics (2019),‌project_info_‌description.cfm?aid=9694688 [].

  56. . For instance, Duke developed a kidney risk algorithm and integrated it into clinical practice for only $217,138. Mark P. Sendak, Suresh Balu & Kevin A. Schulman, Barriers to Achieving Economics of Scale in Analysis of EHR Data, 8 Applied Clin. Informatics 826, 828 (2017). Nevertheless, continuing the theme of local innovation, the developers estimated that scaling the algorithm nationally would cost $38.8 million. Id.

  57. . See Ctrs. Medicare & Medicaid Servs., AI Health Outcomes Challenge (2019), [] (listing Geisinger, Jefferson Health, Mayo Clinic, Northwestern, UVA as health systems receiving awards).

  58. . See I. Glenn Cohen & Michelle M. Mello, Big Data, Big Tech, and Protecting Patient Privacy, 322 J Am. Med. Ass’n 1141 (2019); Kayte Spector-Bagdady, Raymond Hutchinson, Erin O’Brien Kaleba & Sachin Kheterpal, Sharing Health Data and Biospecimens with Industry – A Principle-Driven, Practical Approach, 382 New Eng. J. Med. 2072 (2020). Some of these collaborations have been the subject of significant controversy, particularly where physicians or scientists at nonprofit institutions have obtained equity stakes in profit-driven companies. See, e.g., Charles Ornstein & Katie Thomas, Sloan-Kettering’s Cozy Deal with Start-Up Ignites a New Uproar, N.Y. Times (Sept. 20, 2018), [].

  59. . See Price, supra note 3, at 434.

  60. . See Ctrs. for Medicare & Medicaid Servs., U.S. Dep’t Health & Human Servs., Fiscal Year 2016 Justification of Estimates for Appropriations Committees 109 (2015), [].

  61. . Erin C. Fuse Brown, Consumer Financial Protection in Health Care, 95 Wash. U. L. Rev. 127, 188 (2017).

  62. . Korobkin, supra note 41, at 10, 12–13.

  63. . Meredith Freed, Anthony Damico & Tricia Neuman, A Dozen Facts About Medicare Advantage in 2020, Kaiser Fam. Found. (Jan. 13, 2021), [].

  64. . Elizabeth Hinton, Robin Rudowitz, Lina Stolyar & Natalie Singer, 10 Things to Know About Medicaid Managed Care, Kaiser Fam. Found. (Oct. 29, 2020), [].

  65. . Although as discussed infra in Section III.C, a growing body of scholarship has started to address the role that insurers and reimbursement may play in the health innovation space. See, e.g., Sachs, Prizing Insurance, supra note 10 (describing insurance reimbursement incentives for insurers); Mark A. Lemley, Lisa Larrimore Ouellette & Rachel E. Sachs, The Medicare Innovation Subsidy, 95 N.Y.U. L. Rev. 75 (2020) (describing how Medicare reimbursement creates subsidies for particular types of biomedical innovation); Rebecca S. Eisenberg, Shifting Institutional Roles in Biomedical Innovation in a Learning Healthcare System, 14 J. Inst. Econ. 1139 (2018) (describing the new role of insurers in innovation); Rebecca S. Eisenberg & W. Nicholson Price II, Promoting Healthcare Innovation on the Demand Side, 4 J. L. & Biosciences 3 (2017) (describing innovation by insurers themselves). Payers have also played an indirect role in innovation to the extent that they have performed technology assessments in deciding whether and under what circumstances to pay for particular products, in which the payers consider the clinical evidence supporting the product. See, e.g., Julia R. Trosman, Stephanie L. Van Bebber & Kathryn A. Phillips, Coverage Policy Development for Personalized Medicine: Private Payer Perspectives on Developing Policy for the 21-Gene Assay, 6 J. Oncology Prac. 238 (2010).

  66. . In Section III.C, we discuss the role of insurers in providing reimbursement for hospital-based AI model use.

  67. . At least one large insurance company, Anthem, has developed a policy of denying coverage for emergency room visits that it later deems to have been “unnecessary.” This policy has come under strong criticism, as patients themselves do not always know whether a hospital visit is “necessary” when symptoms are concerning. See, e.g., Sarah Kliff, An ER Visit, a $12,000 Bill – And a Health Insurer That Wouldn’t Pay, Vox (Jan. 29, 2018, 8:00 AM), []; Samantha Raphelson, Anthem Policy Discouraging “Avoidable” Emergency Room Visits Faces Criticism, NPR (May 23, 2018, 4:18 PM), [].

  68. . See, e.g., Bruce Japsen, Health Insurers Hire Thousands of Nurses to Coordinate Care, Forbes (Sept. 25, 2015, 7:57 AM), [].

  69. . Jim Molis, AI Tools Improves Care to Patients with Chronic Conditions – and Savings for Insurers and Employers, Pitt. Bus. Times (May 1, 2020), [].

  70. . Press Release, Highmark Health, Highmark Health Partners with Google Cloud to Raise Standard for Customer and Clinician Engagement in Health (Dec. 17, 2020), [].

  71. . Ziad Obermeyer, Brian Powers, Christine Vogeli & Sendhil Mullainathan, Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations, 366 Science 447, 447 (2019); Quinn Gawronski, Racial Bias Found in Widely Used Health Care Algorithm, NBC News (Nov. 7, 2019, 10:07 AM), [].

  72. . Tom Davenport & Randy Bean, Optum Focuses on AI to Improve Administrative Decisions, Forbes (Oct. 9, 2020, 10:50 AM), [].

  73. . The named insurers are not the only ones to have developed venture funds. For instance, BlueCross BlueShield has also developed a Venture Fund. See Blue Cross and Blue Shield Association, Blue Venture Fund, []. BCBS has also engaged in novel innovation challenges, such as its Data Innovation Challenge which rewarded the winning firms (of over 130 applicants) not with money, but with access to patient data. BlueCross BlueShield Association, Thrive Earlier Detection Wins the BlueCross BlueShield Data Innovation Challenge (2019), [].

  74. . Tracey Walker, Cigna Leverages AI, Managed Healthcare Exec. (July 15, 2019), [].

  75. . Christopher Snowbeck, Optum Ventures Backs Startup That Uses AI To Aid in Medical Diagnosis, Star Trib. (Sept. 26, 2018, 7:40 PM), [].

  76. . About Us, Mindstrong Health, (last visited Nov. 14, 2021).

  77. . Business Development, Mayo Clinic Ventures (2021), [].

  78. . The Rise of Hospital-Backed Venture Capital Funds, Becker’s Hosp. Rev. (Oct. 15, 2014), [].

  79. . Id.

  80. . Pub. L. No. 75-717, 52 Stat. 1040 (1938) (codified as amended in provisions set forth in Title 21 of the U.S. Code).

  81. . See Eisenberg, supra note 10.

  82. . FDA-Approved A.I.-Based Algorithms, Med. Futurist, [].

  83. . See, e.g., Off. Nat’l Coordinator for Health Info. Tech., FDASIA Health IT Report: Proposed Strategy and Recommendations for a Risk-Based Framework 3–4 (2014), [].

  84. . See Andrew W. Torrance & Eric von Hippel, The Right to Innovate, 2015 Mich. St. L. Rev. 793.

  85. . See 21 U.S.C. § 396 (“Nothing in this chapter shall be construed to limit or interfere with the authority of a health care practitioner to prescribe or administer any legally marketed device to a patient for any condition or disease within a legitimate health care practitioner-patient relationship.”); Wendy Teo, FDA and the Practice of Medicine: Looking at Off-Label Drugs, 41 Seton Hall Legis. J. 305 (2017). But see Patricia J. Zettler, Toward Coherent Federal Oversight of Medicine, 52 San Diego L. Rev. 427 (2015) (arguing the distinction between medical products and the practice of medicine is indistinct).

  86. . Richard R. Nelson, Kristin Buterbaugh, Marcel Perl & Annetine Gelijns, How Medical Know-How Progresses, 40 Rsch. Pol’y 1339 (2011).

  87. . See Annetine Gelijns & Nathan Rosenberg, The Dynamics of Technological Change in Medicine, 13 Health Affs. 28 (1994).

  88. . 21 U.S.C. § 331; see also § 321(b) (“The term “interstate commerce means (1) commerce between any State or Territory and any place outside thereof, and (2) commerce within the District of Columbia or within any other Territory not organized with a legislative body.”). Although health and safety regulation is traditionally relegated to the states, the limitation of prohibited activities to interstate commerce gave Congress authority to enact the legislation under Article I, Section 8 of the Constitution: “The Congress shall have Power . . . To regulate Commerce with foreign Nations, and among the several States . . . .” U.S. Const. art. I, § 8, cl. 3.

  89. . These changes in the types of technological innovation taking place over time—moving from a “product” paradigm to a “practice” paradigm, as Professor Nathan Cortez has framed it—challenge FDA’s ability to rely on older statutes to address newer issues. Nathan Cortez, Substantiating Big Data in Health Care, 14.1 I/S: J. L. & Pol’y for Info Soc’y 61, 72–73 (2017).

  90. . Paul D. Clement & Laurence H. Tribe, Laboratory Testing Services, As the Practice of Medicine, Cannot Be Regulated as Medical Devices 10 (2015), [].

  91. . United States v. Regenerative Scis., 741 F.3d 1314, 1320 (D.C. Cir. 2014) (affirming jurisdiction of FDA to enforce FDCA against medical practice that treated patients with a mixture of mesenchymal stem cells extracted from the patients with an antibiotic that had been shipped in interstate commerce); see also United States. v. U.S. Stem Cell Clinic, LLC, 403 F. Supp. 3d 1279, 1283, 1298 n.11, 1300 n.12 (S.D. Fla. 2019), aff’d, 998 F.3d 1302, 1304 (11th Cir. 2021) (affirming jurisdiction of FDA to enforce FDCA against clinic performing stem cell therapies using reagents purchased from out of state).

  92. . § 321(h). In 2016 Congress added an explicit exclusion for “software functions excluded pursuant to section 360j(o).”

  93. . The statute further divides this broad category into different classes with increasing regulatory controls based on the degree of risk they pose. § 360c. Other statutory language includes in the definition of “device” an article which is “(1) recognized in the official National Formulary, or the United States Pharmacopeia, or any supplement to them,” or “(3) intended to affect the structure or any function of the body of man or other animals.” § 321(h). The AI technologies considered in this paper are unlikely to meet either of these alternative prongs of the device definition.

  94. . § 321(h).

  95. . Pub. L. No. 114-255, 130 Stat. 1033 (2016).

  96. . Cures Act § 3060(a), codified at 21 U.S.C. § 360j(o). As amended, the statute excludes from the definition of device a software function that is intended

    (A) for administrative support of a health care facility. . . ; (B) for maintaining or encouraging a healthy lifestyle . . . ; (C) to serve as electronic patient records . . . so long as – (i) such records were created, stored, transferred, or reviewed by health care professionals, or by individuals working under supervision of such professionals . . . ; (D) for transferring, storing, converting formats, or displaying clinical laboratory test or other device data and results . . . unless such function is intended to interpret or analyze clinical laboratory test or other device data, results, and findings; or (E) . . . for the purpose of (i) displaying, analyzing, or printing medical information about a patient or other medical information . . . ; (ii) supporting or providing recommendations to a health care professional about prevention, diagnosis, or treatment of a disease or condition; and (iii) enabling such health care professional to independently review the basis for such recommendations that such software presents . . .


    Congress also amended the statutory definition of “device” to cross-reference these exclusions. See supra note 92.

  97. . See infra note 100 and accompanying text.

  98. . See supra note 92 and accompanying text.

  99. . See, e.g., U.S. Food & Drug Admin., Changes to Existing Medical Software Policies Resulting from Section 3060 of the 21st Century Cures Act: Guidance for Industry and Food and Drug Administration Staff 6–7 (2019) [hereinafter Cures Act Changes Guidance] (previous guidance indicating that FDA intended to exercise enforcement discretion to refrain from regulating certain medical mobile applications designed to promote general fitness and wellness for individuals would be modified to indicate that these applications no longer meet the definition of “device”); id. at 8–11 (previous guidance indicating that FDA intended to exercise enforcement discretion for software functions that enable individuals to interact with their own electronic health records would be modified to indicate that these functions no longer meet the definition of “device”).

  100. . 21 U.S.C. § 360j(o)(3)(A). Notification of the finding and proposed order must be published in the Federal Register with an opportunity for public comment for at least thirty days before it becomes final. § 360j(o)(3)(B).

  101. . § 360j(o)(3)(D).

  102. . Cures Act Changes Guidance, supra note 99, at 13.

  103. . § 360j(o)(1)(E). FDA draft guidance explains:

    Products that acquire an image or physiological signal from the body, or from a sample from the body, or that process or analyze such information, or both, have been regulated for many years as devices when such acquisition, processing, or analyzing is intended for a purpose identified in the statutory device definition.

    U.S. Food & Drug Admin., Clinical Decision Support Software: Draft Guidance for Industry and Food and Drug Administration Staff 10 (2019) [hereinafter CDS Draft Guidance].

  104. . § 360j(o)(1)(E)(i), (ii).

  105. . § 360j(o)(1)(E)(iii).

  106. . CDS Draft Guidance, supra note 103, at 12.

  107. . Sendak et al., supra note 33, at 5.

  108. . Id.

  109. . See, e.g., Nathan Cortez, The Mobile Health Revolution?, 47 U.C. Davis L. Rev. 1173, 1220–22 (2014); Barbara J. Evans & Pilar Ossorio, The Challenge of Regulating Clinical Decision Support Software After 21st Century Cures, 44 Am. J.L. & Med. 244–50 (2018); Efthimios Parasidis, Clinical Decision Support: Elements of a Sensible Legal Framework, 20 J. Health Care L. & Pol’y 183, 191–208 (2019).

  110. . Pub. L. No. 94-295, 90 Stat. 539 (1976).

  111. . U.S. Food & Drug Admin., Draft Guidance for Industry, Food and Drug Administration Staff, and Clinical Laboratories, Framework for Regulatory Oversight of Laboratory Developed Tests (LDTs) 5–7 (2014).

  112. . Id. at 7–8.

  113. . Id.

  114. . U.S. Food & Drug Admin., Discussion Paper on Laboratory Developed Tests (LDTs) (Jan. 13, 2017), [] [hereinafter Discussion LDTs].

  115. . CMS administers the Clinical Laboratory Improvement Amendments (CLIA), and clinical laboratories must obtain certificates of compliance or accreditation under CLIA in order to receive Medicare and Medicaid reimbursement for the testing services they provide. Ctrs. for Medicare & Medicaid Servs., CLIA, []. CLIA primarily regulates laboratories and their procedures, requiring labs to “meet requirements relating to the proper collection, transportation, and storage of specimens,” 42 U.S.C. § 263a(f)(1)(A), and to “use only personnel meeting such qualifications as the Secretary may establish,” id. at § 263a(f)(1)(C), among other things. But CLIA regulations do impose some requirements on the analytical validity of tests themselves, by requiring laboratories to engage in proficiency testing and to set specifications for their tests’ accuracy, precision, and reportable ranges. 42 C.F.R. § 493.1253(b) (2021). See also Rachel E. Sachs, Innovation Law and Policy: Preserving the Future of Personalized Medicine, 49 U.C. Davis L. Rev. 1881, 1891–94 (2016).

  116. . Rebecca S. Eisenberg, Opting into Device Regulation in the Face of Uncertain Patentability, 23 Marquette Intell. Prop. L. Rev. 1 (2019). Laboratories performing diagnostic testing for COVID-19 may want their LDTs to receive FDA authorization, which allows those firms to receive legal immunity under emergency preparedness laws. Steve Usdin, FDA to Stop Reviewing COVID-19 Lab Tests, Raising Concerns in Congress, BioCentury (Oct. 7, 2020, 8:24 PM), [].

  117. . See Discussion LDTs, supra note 114, at 1–2.

  118. . While locked algorithms provide the same outputs given the same input, inputs change as the real world does, which can degrade algorithm performance over time. Sharon E. Davis, Thomas A. Lasko, Guanhua Chen, Edward D. Siew & Michael E. Matheny, Calibration Drift in Regression and Machine Learning Models for Acute Kidney Injury, 24 J. Am. Med. Informatics Ass’n 1052, 1053 (2017).

  119. . See Boris Babic, Sara Gerke, Theodoros Evgeniou & I. Glenn Cohen, Algorithms on Regulatory Lockdown in Medicine, 366 Sci. 1202, 1202–03 (2019).

  120. . See U.S. Food & Drug Admin., Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD): Discussion Paper and Request for Feedback (2019) [hereinafter Proposed Framework for Modifications].

  121. . Id. at 1, 6, 7–14.

  122. . See Int’l Med. Device Regulator’s F., Software as a Medical Device (SaMD): Key Definitions (2013), []; U.S. Food & Drug Admin., Software as a Medical Device (SAMD): Clinical Evaluation, Guidance for Industry and Food and Drug Administration Staff (Dec. 8, 2017), [].

  123. . U.S. Food & Drug Admin., Developing a Software Precertification Program: A Working Model; v.1 (Jan. 2019), []; see also Rachel E. Sachs, Regulating Intermediate Technologies, 37 Yale J. on Reg. 219, 248 (2020).

  124. . Proposed Framework for Modifications, supra note 120, at 12–15.

  125. . Id. at 11–12.

  126. . Press Release, U.S. Food & Drug Admin., FDA Releases Artificial Intelligence/Machine Learning Action Plan (Jan. 12, 2021), [].

  127. . 21 U.S.C. § 360j(o)(3)(A)(i); see supra note 100 and accompanying text.

  128. . 21 U.S.C. § 360c. Class I devices pose the lowest risk and are subject to the lowest level of regulatory controls, with increasing levels of regulatory controls for the higher risk devices in Class II and Class III. See Overview of Device Regulation, U.S. Food & Drug Admin., [].

  129. . E.g., CDS Draft Guidance, supra note 103, at 16 (indicating when FDA intends to exercise enforcement discretion for low-risk software functions intended to provide clinical decision support). See also U.S. Food & Drug Admin., Policy for Device Software Functions and Mobile Medical Applications, Guidance for Industry and Food and Drug Administration Staff 2 (Sept. 27, 2019) (“[T]he FDA intends to apply its regulatory oversight to only those software functions that are medical devices and whose functionality could pose a risk to a patient’s safety if the device were to not function as intended.”).

  130. . Id. at 16–17, 20–21. Examples in the draft guidance of functions that are intended to inform clinical management for non-serious situations or conditions include a machine-learning algorithm for which the logic and inputs are not explained that alerts health care providers to triggers that may indicate cholesterol management issues.

  131. . Sendak et al., supra note 33, at 6.

  132. . Telephone Interview with Anonymous Head of an Academic Medical Center’s Machine Learning Program (Dec. 30, 2020); E-mail from Anonymous Member of a Major Academic Medical Center’s Machine-Learning Implementation Committee (Aug. 24, 2020) (on file with author).

  133. . Id.

  134. . Indeed, members of the World Trade Organization may now be required to apply the same rules of patent law to all fields under Article 27(1) of the Agreement on Trade-Related Aspects of Intellectual Property Rights, Annex 1C to the Marrakesh Agreement Establishing the World Trade Organization, Apr. 15, 1994, 33 I.L.M. 1125 (TRIPS Agreement).

  135. . Luigi Orsenigo & Valerio Sterzio, Comparative Study of the Use of Patents in Different Industries (Knowledge, Internationalization & Tech. Stud., Universita Bocconi, Working Paper No. 33/2010, 2010) Comparative_Study_of_the_Use_of_Patents_in_Different_Industries/links/00b4953a2d1294b2e4000000.pdf.

  136. . See Dan L. Burk & Mark A. Lemley, Policy Levers in Patent Law, 89 Va. L. Rev. 1575 (2003).

  137. . See, e.g., Iain Cockburn & Genia Long, The Importance of Patents to Innovation: Updated Cross-Industry Comparisons with Biopharmaceuticals, 25 Expert Op. on Therapeutic Patents 739 (2015); Bronwyn H. Hall & Megan MacGarvie, The Private Value of Software Patents, 39 Rsch. Pol’y 994 (2009); Stuart J.H. Graham, Robert P. Merges, Pam Samuelson & Ted Sichelman, High Technology Entrepreneurs and the Patent System: Results of the 2008 Berkeley Patent Survey, 24 Berkeley Tech. L.J. 1255, 1286 (2009).

  138. . Telephone Interview with Yindalon Aphinyanaphongs, Dir., Clinical Predictive Analytics at NYU Langone Health, (Aug. 10, 2020); Telephone Interview with Mark Sendak, Population Health & Data Sci. Lead, Duke Inst. For Health Innovation (July 19, 2021).

  139. . Cf. Kevin Emerson Collins, Copyright and the Customization Effect, 56 Wake Forest L. Rev. 197 (2021) (arguing that customized creative production needs less intellectual property protection).

  140. . See World Intellectual Property Organization, WIPO Technology Trends 2019: Artificial Intelligence 13–14 (2019) (noting that AI-related patenting is growing rapidly, with machine learning patent filings increasing at an average annual rate of 28% since 2013 and using patent data analytics to identify research trends in AI); Howard Read, Artificial Intelligence and Machine Learning in Healthcare: An Intellectual Property Perspective, Appleyard Lees (Dec. 13, 2019), [] (noting a “surge in filings of patent applications by companies that have not traditionally been associated with the healthcare sector” such as Google, Microsoft, and Apple).

  141. . Bilski v. Kappos, 561 U.S. 593 (2010); Mayo Collaborative Servs. v. Prometheus Labs., 566 U.S. 66 (2012); Ass’n for Molecular Pathology v. Myriad Genetics, 569 U.S. 576 (2013); Alice Corp. v. CLS Bank Int’l, 573 U.S. 208 (2014).

  142. . E.g., Tabrez Y. Ebrahim, Data-Centric Technologies: Patents and Copyright Doctrinal Disruptions, 43 Nova L. Rev. 287 (2019).

  143. . E.g., Dan L. Burk, AI Patents and the Self-Assembling Machine, 105 Minn. L. Rev. Headnotes 301 (2021).

  144. . U.S. Patent & Trademark Off., Public Views on Artificial Intelligence and Intellectual Property Policy 1 (Oct. 2020), [].

  145. . Mayo, 566 U.S. at 82. See also Bilski, 561 U.S. 593; Molecular Pathology, 569 U.S. 576; Alice, 573 U.S. 208. The revived limitations claim authority from a number of older decisions including Funk Bros. Seed Co. v. Kalo Inoculant, 333 U.S. 127 (1948); Gottschalk v. Benson, 409 U.S. 63 (1972); and Parker v. Flook, 437 U.S. 584 (1978). Cf. Jeffrey A. Lefstin, The Three Faces of Prometheus: A Post-Alice Jurisprudence of Abstractions, 16 N.C. J.L. & Tech 647, 669 (2015) (characterizing recent decisions as the Court “turn[ing] its back on the traditional framework for patent eligibility” that “drew the boundary of patent-eligibility at practical application” in favor of a new requirement of “inventive application.”).

  146. . See Jeffrey A. Lefstin, Peter S. Mennell & David O. Taylor, Final Report of the Berkeley Center for Law & Technology Section 101 Workshop: Addressing Patent Eligibility Challenges, 33 Berkeley Tech. L.J. 551, 581–591 (2018) (summarizing views expressed at a workshop including industry representatives, legal practitioners, legal scholars, and policymakers); Rebecca S. Eisenberg, Diagnostics Need Not Apply, 21 B.U. J. Sci. & Tech. L. 256 (2015).

  147. . 566 U.S. at 66.

  148. . U.S. Patent No. 6,355,623 (filed Mar. 12, 2002).

  149. . 566 U.S. at 72.

  150. . 573 U.S. at 208.

  151. . U.S. Patent No. 5,970,479 (filed Oct. 19, 1999).

  152. . Alice, 573 U.S. at 217–18 (quoting Mayo, 566 U.S. at 72–73). Critics say this test improperly conflates patent eligibility with satisfaction of patent law standards and creates considerable uncertainty as to the types of inventions that are patent eligible, particularly in the life sciences and information technology. Kevin Emerson Collins, Bilski and the Ambiguity of “An Unpatentable Abstract Idea”, 15 Lewis & Clark L. Rev. 37 (2011); John M. Golden, Flook Says One Thing, Diehr Says Another: A Need for Housecleaning in the Law of Patentable Subject Matter, 82 Geo. Wash. L. Rev. 1765 (2014); Eisenberg, supra note 146.

  153. . Robert Sachs, Alice: Benevolent Despot or Tyrant? Analyzing Five Years of Case Law Since Alice v. CLS Bank: Part I, IP Watchdog (Aug. 29, 2019), [].

  154. . For a thoughtful and comprehensive review of recent cases, see Paul R. Gugliuzza, The Procedure of Patent Eligibility, 97 Tex. L. Rev. 571, 581–91 (2018) (finding that courts often invalidate patent for lack of patentable subject matter at the pleading stage without developing an evidentiary record).

  155. . Disagreement and lament appear in multiple opinions from members of the Federal Circuit concurring or dissenting from the denial of rehearing en banc in Athena Diagnostics v. Inc. v. Mayo Collaborative Servs., LLC, 927 F.3d 1333 (Fed. Cir. July 3, 2019) (denying rehearing en banc in Athena Diagnostics, Inc. v. Mayo Collaborative Servs. LLC, 915 F.3d 743 (Fed. Cir. 2019)). See, e.g., Athena, 927 F.3d at 1335 (opinion of Lourie, J., joined by Reyna & Chen J.J., concurring in the denial of the petition for rehearing en banc) (“If I could write on a clean slate, I would write as an exception to patent eligibility, as respects natural laws, only claims directed to the natural law itself . . . I would not exclude uses or detection of natural laws . . . But we do not write here on a clean slate; we are bound by Supreme Court precedent.”); id. at 1337 (opinion of Hughes, J., joined by Prost, C.J. & Taranto, J., concurring in the denial of the petition for rehearing en banc) (“I agree that the language in Mayo, as later reinforced in Alice, forecloses this court from adopting an approach or reaching a result different from the panel majority’s. I also agree, however, that the bottom line for diagnostics patents is problematic. But this is not a problem that we can solve. As an inferior appellate court, we are bound by the Supreme Court.”).

  156. . Lefstin, Mennell & Taylor, supra note 146; Shahrokh Falati, Patent Eligibility of Disease Diagnosis, 21 N.C. J.L. & Tech. 63 (2020); David O. Taylor, ‘Amending Patent Eligibility’, 50 U.C. Davis L. Rev. 2149 (2017); Robert Sachs, Alice: Benevolent Despot or Tyrant? Analyzing Five Years of Case Law Since Alice v. CLS Bank: Part II, IP Watchdog (Sept. 3, 2019), []; Mateo Aboy, Cristina Crespo, Kathleen Liddell, Timo Minssen & Johnathan Liddicoat, Mayo’s Impact on Patent Applications Related to Biotechnology, Diagnostics and Personalized Medicine, 37 Nature Biotechnology 513, 514–16 (2019).

  157. . Mateo Aboy et al., One Year After Vanda, Are Diagnostics Patents Transforming into Methods of Treatment to Overcome Mayo-Based Rejections?, 38 Nature Biotechnology 279, 281–82 (2020).

  158. . The Federal Circuit and the PTO continue to consider arguments to apply the Alice/Mayo two-part test narrowly. See, e.g., Vanda Pharms. Inc. v. West-Ward Pharms. Int’l Ltd., 887 F.3d 1117 (Fed. Cir. 2018) (claims reciting method of treatment based on results of diagnostic test are not excluded from patent eligibility notwithstanding that treatment steps are conventional), cert. petition filed, call for views of solicitor general sub nom. Hikma Pharms. v. Vanda Pharms. (Mar. 18, 2019); cert. denied, 140 S. Ct. 911 (2020); Nat. Alts. Int’l, Inc. v. Creative Compounds, LLC, 918 F.3d 1338 (Fed. Cir. 2019) (method of treatment claims patent eligible even though method uses natural products and treatment steps involve conventional, well-known activity). The Supreme Court has resisted pleas to revisit patent subject matter eligibility, Hikma Pharms Inc. v. Vanda Pharms., 140 S. Ct. 911 (2020); HP v. Berkheimer, 140 S. Ct. 911 (Mem) (2020). Though the Solicitor General has noted that recent cases “have fostered substantial uncertainty” and that “[t]he confusion created by this Court’s recent Section 101 precedents warrants review in an appropriate case.” Brief for the United States as Amicus Curiae Supporting Respondents at 8, Hikma Pharms. v. Vanda Pharms., 140 S. Ct. 911 (2020) (No. 18-817) (Dec. 19, 2019), [].

  159. . Mateo Aboy et al., How Does Emerging Patent Case Law in the US and Europe Affect Precision Medicine?, 37 Nature Biotechnology 1118, 1121 (2019).

  160. . 35 U.S.C. § 112(a), (b).

  161. . W. Nicholson Price II, Big Data, Patents, and the Future of Medicine, 37 Cardozo L. Rev. 1401, 1427–32 (2015); See also Burk, supra note 143, at 2 (“[I]n many cases the solutions developed for the patenting of biotechnological inventions appear to provide ready answers to the concerns raised regarding patents and AI technologies.”).

  162. . Price, supra note 161, at 1429.

  163. . Id. at 1430–32.

  164. . Id. at 1432–36.

  165. . Id.

  166. . As noted above, the biomedical industry is not the only relevant baseline; incentives provided by patents for medical AI may more closely resemble those for software more generally, where patents have long been of questionable value. Julie E. Cohen & Mark A. Lemley, Patent Scope and Innovation in the Software Industry, 89 Cal. L. Rev. 1, 5–6 (2001); Graham et al., supra note 137, at 1262.

  167. . For some explicitly diagnostic AI technologies, insurance reimbursement may be available. However, those technologies are not the focus of this Article. They are much more likely to fit into the classic medical device paradigm and likely also are subject to FDA review.

  168. . In the case of prescription drugs, insurers are often compelled by law to provide such reimbursement. Rachel E. Sachs, Delinking Reimbursement, 102 Minn. L. Rev. 2307, 2316–17 (2018).

  169. . See, e.g., Sachs, Prizing Insurance, supra note 10, at 178; Rebecca S. Eisenberg & W. Nicholson Price II, Promoting Healthcare Innovation on the Demand Side, 4 J.L. & Biosciences 3, 5 (2017); Mark A. Lemley, Lisa Larrimore Ouellette & Rachel E. Sachs, The Medicare Innovation Subsidy, 95 N.Y.U. L. Rev. 75, 105–07 (2020); Kevin Outterson, The Legal Ecology of Resistance: The Role of Antibiotic Resistance in Pharmaceutical Innovation, 31 Cardozo L. Rev. 613, 645–55 (2010); Arti K. Rai, The Ends of Intellectual Property: Health as a Case Study, 70 L. & Contemp. Probs. 125, 128–29 (2007); Benjamin N. Roin, Intellectual Property Versus Prizes: Reframing the Debate, 81 U. Chi. L. Rev. 999, 1012–13 (2014).

  170. . Sachs, Prizing Insurance, supra note 10, at 178. See also generally, e.g., Michael Abramowicz, Perfecting Patent Prizes, 56 Vand. L. Rev. 115 (2003); Nancy Gallini & Suzanne Scotchmer, Intellectual Property: When Is It the Best Incentive System?, 2 Innovation Pol’y & Econ. 51 (2002); see also Steven Shavell & Tanguy van Ypersele, Rewards Versus Intellectual Property Rights, 44 J. L. & Econ. 525, 528 (2001) (cataloging the literature).

  171. . Daniel J. Hemel & Lisa Larrimore Ouellette, Beyond the Patents — Prizes Debate, 92 Tex. L. Rev. 303, 307–08, 348 (2013).

  172. . These pressures are less acute for non-health goods, which typically require fewer resources to develop in the first instance (as FDA review process is both costly and time consuming) and which are typically inexpensive enough to enable consumers to purchase them directly. Americans might be able to save up for a new or used smartphone—a one-time purchase costing several hundred dollars—but cannot afford more than a million dollars per year for a lifesaving medication. See, e.g., Katie Thomas & Reed Abelson, The $6 Million Drug Claim, N.Y. Times (Aug. 25, 2019), [].

  173. . Medicare Prescription Drug, Improvement, and Modernization Act of 2003, Pub. L. No. 108-173, 117 Stat. 2066 (codified in scattered sections of 26 and 42 U.S.C.). Part D was not created until nearly forty years after the passage of the initial Medicare statute. Paul Starr, The Social Transformation of American Medicine 368–70 (1982).

  174. . Kaiser Fam. Found., Prescription Drug Trends 2010 5 (May 2010) (“Prior to January 1, 2006, . . . about one-quarter (27%) of seniors age 65 and older, and one-third of poor (34%) and near-poor (33%) seniors, had no drug coverage in 2003.”); see also Dana Gelb Safran et al., Prescription Drug Coverage and Seniors: Findings from a 2003 National Survey, Health Affs. W5-152, W5-160 (Apr. 19, 2005),‌content/early/‌2005/04/‌19/‌hlthaff.w5.152.citation [].

  175. . Richard G. Frank & Joseph P. Newhouse, Should Drug Prices Be Negotiated Under Part D of Medicare? And if so, How?, 27 Health Affs. 33, 34, 36–37 (2008).

  176. . See Margaret E. Blume-Kohout & Neeraj Sood, The Impact of Medicare Part D on Pharmaceutical R&D 12–13 (Nat’l Bureau of Econ. Rsch., Working Paper No. 13857, 2008). But see David Dranove, Craig Garthwaite & Manuel Hermosilla, Pharmaceutical Profits and the Social Value of Innovation 6–7 (Nat’l Bureau of Econ. Rsch., Working Paper No. 20212, 2014) (qualifying the findings of Blume-Kohout and Sood by noting that truly innovative activity takes longer to emerge).

  177. . See, e.g., Amy Finkelstein, Static and Dynamic Effects of Health Policy: Evidence from the Vaccine Industry, 119 Q.J. Econ. 527, 528 (2004); Wesley Yin, Market Incentives and Pharmaceutical Innovation, 27 J. Health Econ. 1060, 1061 (2008).

  178. . See Thompson, supra note 42.

  179. . One hospital uses machine learning to identify un-coded reimbursable elements in electronic health record notes and flags them for review by manual reviewers. Telephone Interview with Anonymous Head of an Academic Medical Center’s Machine Learning Program (Dec. 30, 2020).

  180. . See, e.g., Sylvia M. Burwell, Setting Value-Based Payment Goals—HHS Efforts to Improve U.S. Health Care, 372 New Eng. J. Med. 897, 897 (2015); Press Release, U.S. Dept. of Health & Human Servs., HHS to Deliver Value-Based Transformation in Primary Care (Apr. 22, 2019), [].

  181. . See, e.g., Teresa M. Waters et al., Effect of Medicare’s Nonpayment for Hospital-Acquired Conditions: Lessons for Future Policy, 175 JAMA Internal Med. 347 (2015).

  182. . Duke Today Staff, Duke Health Licenses Technology Aimed to Reduce Sepsis in Hospitals, Duke Today Med. (July 3, 2019), [].

  183. . Ctrs. for Medicare & Medicaid Servs., Hospital Readmissions Reduction Program (HRRP) (Jan. 6, 2020), [].

  184. . Telephone Interview with Anonymous Head of an Academic Medical Center’s Machine Learning Program (Dec. 30, 2020).

  185. . Sachs, supra note 168, at 2309.

  186. . What Is Prior Authorization and How Does the Process Work?, CIGNA (2018), [].

  187. . See supra Section III.A.

  188. . Nick Paul Taylor, CMS Rejects 1st Colorectal Cancer Blood Test, Tweaks Path for Exact, Guardant and Others, HealthcareDive (Jan. 20, 2021), [].

  189. . Eisenberg & Price II, supra note 65, at 3; Eisenberg, supra note 10, at 374.

  190. . See Obermeyer et al., supra note 71.

  191. . An instructive parallel is laboratory-developed diagnostic tests. True, FDA exercises enforcement discretion against the makers of laboratory-developed tests at least in part because those tests were traditionally relatively simple. But it is also likely that some institutions are willing to develop and use their own tests in reliance on enforcement discretion when that development might not be cost-effective if coupled with an FDA clearance process. See supra text accompanying notes 110–117.

  192. . W. Nicholson Price II, Big Data, Patents, and the Future of Medicine, 37 Cardozo L. Rev. 1401, 1411–17 (2016).

  193. . W. Nicholson Price II, Medical AI and Contextual Bias, 33 Harv. J.L. & Tech. 65, 81–83 (2020) [hereinafter Price, Contextual Bias]; see also von Hippel, Democratizing, supra note 7, at 8 (“When information is sticky, innovators tend to rely largely on information they already have in stock.”). At the margins, these incentives may exacerbate existing pressures in favor of hospital merger and acquisition activity.

  194. . Telephone Interview with Anonymous Veterans Aff. Official Involved in Informatics (June 25, 2020).

  195. . See, e.g., Announcement of Requirements and Registration for “Artificial Intelligence Health Outcomes Challenge”, Ctrs. for Medicare & Medicaid Servs. (Aug. 13, 2020), []; BlueCross BlueShield Data Innovation Challenge, Blue Cross Blue Shield Ass’n, (Aug. 13, 2019), [].

  196. . Jennifer Couzin-Frankel, Artificial Intelligence Could Revolutionize Medical Care. But Don’t Trust It to Read Your X-ray Just Yet, Sci. (Jun. 17, 2019),

  197. . Governmental efforts like the All of US cohort are an attempt to fill the gap. See Joshua C. Denny et al., The “All of Us” Research Program, 381 New Eng. J. Med. 668, 668 (2019). Theoretically, large-scale organizations could self-organize to promote data-sharing, but there are structural barriers to such efforts, including HIPAA and other privacy protections. See W. Nicholson Price II, Risk and Resilience in Health Data Infrastructure, 16 Colo. Tech. L.J. 65 (2017) [hereinafter Price, Risk and Resilience]. If large organizations can make substantial progress using only their own data—even if this progress falls short of what consolidated datasets might permit—they face lower incentives to try to overcome those barriers.

  198. . Cf. Jenna Wiens, John Guttag & Eric Horvitz, A Study in Transfer Learning: Leveraging Data from Multiple Hospitals to Enhance Hospital-Specific Predictions, 21 J. Am. Med. Informatics Ass’n 699, 699 (2014) (finding better performance of C. diff. infections when a model was trained on data from multiple hospitals).

  199. . See W. Nicholson Price II & Arti K. Rai, Clearing Opacity Through Machine Learning, 106 Iowa L. Rev. 775 (2021).

  200. . See Price, Contextual Bias, supra note 193, at 90–98.

  201. . von Hippel, Democratizing, supra note 7, at 133–46.

  202. . See supra notes 38–39 and accompanying text.

  203. . See supra note 194.

  204. . See generally Ctrs. for Medicare & Medicaid Servs., Rural-Urban Disparities in Health Care in Medicare (Nov. 2020).

  205. . See, e.g., Price & Rai, supra note 199, at 801–09 (describing mandated disclosure mediated by (surprise!) the levers of patent law, FDA regulation, and insurer reimbursement).

  206. . von Hippel, Democratizing, supra note 7, at 8 (“One consequence of the information asymmetry between users and manufacturers is that users tend to develop innovations that are functionally novel, requiring a great deal of user-need information and use-context information for their development. In contrast, manufacturers tend to develop innovations that are improvements on well-known needs and that require a rich understanding of solution information for their development.”).

  207. . See, e.g., John E. Wennberg, Shannon Brownlee, Elliot S. Fisher, Jonathan S. Skinner & James N. Weinstein, Dartmouth Inst. for Health Pol’y & Clinical Prac., An Agenda for Change: Improving Quality and Curbing Health Care Spending: Opportunities for the Congress and the Obama Administration (2008), [] (describing practice variations and pattern changes); Off. Nat’l Coordinator for Health Info. Tech., Connecting Health and Care for the Nation: A 10-Year Vision to Achieve an Interoperable Health IT Infrastructure (2014), [ (describing interoperability and data fragmentation challenges).

  208. . Thompson, supra note 42.

  209. . Karen Hao, How an AI Tool for Fighting Hospital Deaths Actually Worked in the Real World, MIT Tech. Rev. (October 2, 2020), [

  210. . In a highly stylized account of how patents promote innovation, the patent system theoretically directs innovators to make the most socially beneficial innovations by linking rewards to the market value of patented goods—but as we have discussed, the incentives from the patent system appear to be less salient in this context. In addition, a substantial literature catalogs the limits of the ability of patents to drive socially beneficial innovation. E.g., Amy Kapczynski & Talha Syed, The Continuum of Excludability and the Limits of Patents, 122 Yale L.J. 1900 (2013); Sachs, Prizing Insurance, supra note 10; Amy Kapczynski, The Cost of Price: Why and How to Get Beyond Intellectual Property Internalism, 59 UCLA L. Rev. 970 (2012), W. Nicholson Price II, The Cost of Novelty, 120 Colum. L. Rev. 769 (2020). Additionally, to the extent that AI tools may help reduce costs (even if not increasing quality irrespective of costs), there is still significant social value in their development. Currently, there are relatively few legal incentives to invest in cost-reducing innovations, but the combination of policy tools in the AI space might encourage their development here.

  211. . von Hippel, Democratizing, supra note 7, at 33–44.

  212. . See I. Glenn Cohen & Michelle M. Mello, Big Data, Big Tech, and Protecting Patient Privacy, 322 J. Am. Med. Ass’n 1141, 1141–42 (2019).

  213. . For a review of the literature, see Nikolaus Franke & Christian Lüthje, User Innovation 10–13 (Jan. 2020), [

  214. . See Eric von Hippel, Lead Users: A Source of Novel Product Concepts, 32 Mgmt. Sci. 791–805 (1986).

  215. . Torrance & von Hippel, supra note 84, at 45–74.

  216. . See supra note 48 and accompanying text.

  217. . Real-Time Clinical Insights for Physicians and CDI Teams, 3M (2019),

  218. . Real Results: A Profile of Eight Organizations Boosted by the 3M™ 360 Encompass™ System, 3M Health Info. Sys. (2019),

  219. . Medical Payment Integrity, Kirontech, [

  220. . Price II, supra note 54; see also Bridge to Artificial Intelligence (Bridge2AI), NIH, [ (announcing a grant program to develop datasets, software and standards, best practices, and training materials for medical AI development).

  221. . To be sure, algorithms that have the highest impact on patient care are those FDA is most likely to regulate, and even if software does not normally fall within the definition of a medical device under the Cures Act, FDA can determine that the software needs to be regulated anyway. See 21 U.S.C. § 360j(o)(3)(A).

  222. . See Couzin-Frankel, supra note 196.

  223. . John R. Zech et al., Variable Generalization Performance of a Deep Learning Model to Detect Pneumonia in Chest Radiographs: A Cross-Sectional Study, PLOS Med. 3 (Nov. 6, 2018) [

  224. . Id. at 11.

  225. . Id. at 13.

  226. . See supra text at notes 120–122.

  227. . Dong Wook Kim, Hye Young Jang, Kyung Won Kim, Youngbin Shin & Seong Ho Park, Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers, 20 Korean J. Radiology 405, 405 (2019) (finding that only 6% of 516 published studies for medical AI image analysis included external validation).

  228. . Price, supra note 3, at 433.

  229. . The health system as a whole is generally poor at monitoring holistic patient outcomes and learning from them. The idea of a learning health system aims to fix this problem. See Digital Infrastructure for the Learning Health System 2–3 (Claudia Grossman, Brian Powers & J. Michael McGinnis eds., 2011); Harlan M. Krumholz, Big Data and New Knowledge in Medicine: The Thinking, Training, and Tools Needed for a Learning Health System, 33 Health Affs. 1163, 1164–69 (2014).

  230. . Ziad Obermeyer, Brian Powers, Christine Vogeli & Sendhil Mullainathan, Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations, 366 Sci. 447, 447 (2019); Heidi Ledford, Millions of Black People Affected by Racial Bias in Health-Care Algorithms, Nature (Oct. 24, 2019), [] (identifying the developer as Optum).

  231. . Ledford, supra note 230, at 608.

  232. . See supra note 101 and accompanying text.

  233. . And the group might also have a vested interest against giving these products an endorsement.

  234. . Clinical Laboratory Improvement Amendments (CLIA), Ctrs. for Medicare & Medicaid Servs., [].

  235. . See id.; see also supra note 115.

  236. . See supra notes 121–126 and accompanying text.

Published by W. Nicholson Price II, Rachel E. Sachs & Rebecca S. Eisenberg

WNP is Professor of Law, University of Michigan Law School; Core Partner, Centre for Advanced Studies in Biomedical Innovation Law at the University of Copenhagen; and Co-PI, Project on Precision Medicine, AI, and the Law at the Petrie-Flom Center at Harvard Law School. RES is Treiman Professor of Law, Washington University in St. Louis School of Law. RSE is Robert and Barbara Luciano Professor of Law, University of Michigan Law School.

%d bloggers like this: