On September 18, 2024, the Texas Attorney General announced a settlement with Pieces Technologies, a Dallas-based artificial intelligence technology company that sells software to hospitals and other medical facilities.
Here’s why this settlement is important:
This is the first settlement by a state Attorney General for the deceptive marketing of AI software. Although the settlement does not include a monetary payment, the Texas Attorney General imposed strict requirements on the future marketing of Pieces’ AI software. This settlement is an important reminder that states do not need new laws to regulate the use of AI. AI software company accused of overstating software’s accuracy
According to the settlement, Peace designed a generative AI product to assist medical staff with patient care, including by drafting summaries, notes and clinical notes. Peace provided its software to at least four hospitals in Texas, where it allegedly made false statements about the product’s accuracy and safety.
In particular, the Texas Attorney General alleged that Pieces advertised metrics showing that its product was “highly accurate” and had an error rate or “rate of severe hallucinations” of “less than 1 in 100,000.” However, the Texas Attorney General argued that these metrics were arguably inaccurate and deceptive, violating the Texas Consumer Protection Act.
Settlement Requires Restrictions on Continuing Marketing and Disclosures
Pieces has agreed to a Voluntary Compliance Assurance (AVC) which imposes strict requirements on its future marketing, including:
All Pieces advertising will clearly and conspicuously disclose the definitions of any metrics used to support accuracy claims and how the metrics are calculated. Pieces is prohibited from making false, misleading or unsubstantiated representations about the accuracy, functionality, purpose or other characteristics of its products. Pieces will clearly and conspicuously disclose in writing to all current and prospective customers any known or reasonably perceptible harmful or potentially harmful uses or misuses of its products.
These disclosures are intended to help hospital staff understand the extent to which they can rely on Pieces’ generative AI products. Under Texas law, a violation of the AVC’s obligations is clear evidence of a violation of the Texas Consumer Protection Act.
Settlement has important implications for AI companies
This settlement is an important reminder that states do not need new laws to regulate the use of AI because consumer protection, privacy, and anti-discrimination laws already apply to AI systems. In June, the Texas Attorney General announced a new data privacy and security initiative focused on data privacy and AI enforcement under several old and new statutes, and an exhaustive enforcement exercise has already occurred involving over 100 companies. The Massachusetts Attorney General’s Office issued guidance in April warning that misrepresenting the reliability, performance method, safety, or condition of an AI system, as well as making unsubstantiated claims about its capabilities, violate Massachusetts consumer protection laws.
Manatt will continue to track these developments closely as state regulators provide additional guidance and other states, including California, move forward with AI-related legislation that will have a major impact on healthcare.
