Effective July 5, 2023, New York City now requires employers or employment agencies that use automated employment decision tools (“AEDTs”) to submit those AEDTs to periodic bias audits, to make information about the bias audit publicly available, and to provide certain notices to job candidates or employees. On June 29, 2023, the New York City Department of Consumer and Worker Protection (DCWP) released a set of frequently asked questions (FAQs) to provide guidance about the new law.[1] A key takeaway from the law, known as New York Local Law 144, is that employers and employment agencies who use AEDTs must adhere to audit and disclosure requirements. Failure to do so runs the risk of prospective enforcement actions.
What is an AEDT?
An AEDT replaces or “substantially assist[s]” “discretionary decision making” in hiring or other employment-related decisions by relying on factors generated by a machine (likely using artificial intelligence algorithms). The FAQs and Rules provide detailed definitions of machine learning, statistical modeling, data analytics and artificial intelligence that generate predictions of a job candidate’s “fit or likelihood of success” based on inputs and that therefore qualify as AEDTs.
When Does the Law Apply?
The statute applies to employees who will work (at least part time) in New York City or who will work remotely outside of the New York City when the location associated with the position is an office in New York City. It also applies to employment agencies located in New York City. The statute applies to the use of AEDTs to assist with hiring once a candidate has actually applied for a specific position.
Bias Audits
The law requires employers or employment agencies that use AEDTs to submit the AEDT to a bias audit prior to use and at least once per year thereafter. The bias audit must be conducted by an independent auditor. The audit must include calculations of selection rates and the impact ratio for each race/ethnicity and sex category that must be reported to the Equal Employment Opportunity Commission (“EEOC”), Rules, § 5-300. These categories are: White, Native Hawaiian or other Pacific Islander, American Indian or Native Alaskan, Asian, Hispanic or Latino, and Black or African American. A bias audit must:
- Be conducted within a year of the most recent bias audit
- Calculate for each category:
- The “selection rate,” which is “the rate at which individuals in a category are either selected to move forward in the hiring process or assigned a classification by an AEDT.” This is done by determining certain percentages, called the “impact ratio” and the “scoring rate” (as calculated by the AEDT).
- “Ensure” that the bias audit “separately calculate[s] the impact of the AEDT on”
- Sex categories
- Race/ethnicity categories (as defined above)
- “Intersectional categories of sex, ethnicity, and race.”
Rules, § 5-300.
The Rules provide a detailed sample audit. The sample audit explains that a “bias audit is necessary even though the employer is not using the AEDT to make the final decision, but only to screen at an early point in the application process.” Rules, § 5-301. At bottom, the bias audit statistically correlates the AEDT’s selection rates to the various race, sex, ethnicity and intersectional categories identified above.
Data Requirements
Identifying the data sources for the AEDT can present significant challenges. A bias audit must rely on the employer’s or employment agency’s own historical data collected from the employer’s or employment agency’s own use of the AEDT. The employer or employment agency may not have historical data if, for example, it has never used the AEDT before. In that situation, the employer or employment agency may submit historical data from another employer’s or employment agency’s use of the AEDT. Rules, § 5-302.
Data Interpretation
What is the point of the collection of all of this data? When does the data raise the possibility of discriminatory employment practices? The FAQs admit that “DCWP has not set a specific requirement for statistical significance.” Similarly, there are no “set requirements” for test data “[t]o allow for flexibility and development of best practices in this rapidly developing field.”
Publication and Notice Requirements
Before an employer or employment agency may use an AEDT it must publish (for example on its website) a notice that it is using an AEDT and the job qualifications or characteristics the AEDT will assess.
The required notice “must include instructions for how an individual can request an alternative selection process or a reasonable accommodation under other law,” however “[n]othing in this subchapter requires an employer or employment agency to provide an alternative selection process.” Rules, § 5-304. Whether the failure to provide an alternative selection process might, under certain circumstances, violate other laws is, for the moment, undecided.
Employers and employment agencies using AEDTs also must publish a summary of the results of the most recent bias audit.
Implications for Employers
Compliance with the new law is complex and burdensome. Given the steep requirements and use of technology tools, a “cottage industry” of bias auditors may emerge. Some employers and employment agencies who currently use or are considering using AEDTs may decide to avoid doing so in local jurisdictions such as New York City that regulate AEDTs. Nonetheless, use of AEDTs is attracting attention nationally as well. The EEOC has already launched an initiative “to ensure that the use of software, including artificial intelligence (AI), machine learning, and other emerging technologies used in hiring and other employment decisions comply with the federal civil rights laws that the EEOC enforces.” U.S. Equal Employment Opportunity Commission, Artificial Intelligence and Algorithmic Fairness Initiative, https://www.eeoc.gov/ai.
Even for employers and employment agencies who obtain a bias audit in compliance with the City law, questions remain. Are certain selection rates, scoring rates and/or impact ratios statistical evidence of discriminatory employment practices? If so, what statistical results will be deemed discriminatory? What evidence will employers or employment agencies be able to use to justify their employment practices if statistical evidence of discrimination is found? The burdens of generating a compliant audit may be just the beginning of an employer’s challenges.
Employers should discuss these and related issues with their employment counsel.
* * *
[1] Unless otherwise indicated, all quotations below are from the FAQs or Rules of City of New York, Subch. T (“Rules”).
Carter Ledyard & Milburn LLP uses Client Advisories to inform clients and other interested parties of noteworthy issues, decisions and legislation which may affect them or their businesses. A Client Advisory does not constitute legal advice or an opinion. This document was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. © 2023 Carter Ledyard & Milburn LLP.