The state Comptroller found two complaints in twenty-four months. Cornell's team found 14 audit reports among 267 employers. Both measurements hit the same wall: a disclosure rule whose scope is set by the discloser produces a record that cannot tell silence from exemption.
Between July 2023 and June 2025, New York City's Department of Consumer and Worker Protection received two complaints about employers using automated hiring tools without the bias audit the city requires. Two, in twenty-four months. The count comes from the New York State Comptroller, whose auditors spent that window's records deciding whether the department had "designed and implemented an effective system to enforce compliance with Local Law 144." Their answer, published on December 2, 2025, was that the department's complaint process "is ineffective in ensuring that all complaints related to non-compliance with LL144 are routed to DCWP."
The auditors then did the thing a complaint count cannot do for itself: they went and looked. The department had surveyed 32 companies about their use of these tools and found one non-compliance issue. The auditors examined the same 32 companies and found 17 potential instances. And the department's own words for the limit it works under, quoted in the audit, are these: "If an employer does not take these steps, it's difficult to identify non-compliance."
That sentence is the subject of this piece. It sounds like an admission of under-resourcing. It is a description of the law's design, and it means the near-empty record the city has built cannot be read as good news or as bad.
Local Law 144 of 2021 prohibits employers and employment agencies from using an automated employment decision tool unless the tool "has been subject to a bias audit within one year of the use of the tool, information about the bias audit is publicly available, and certain notices have been provided to employees or job candidates." Enforcement began on July 5, 2023. The penalties are small per unit and multiply by time: not more than $500 for a first violation, $500 to $1,500 for each subsequent one, and each day a tool is used in violation counts as a separate violation.
Read the requirement again and notice what kind of law it is. It does not tell employers to do something to candidates. It tells them to publish something about themselves. The audit summary must be public. The notice must be given. Compliance leaves a paper trail by definition, which means the law creates its own public record, and anyone with a browser can go and count what is in it.
Somebody did. In the autumn of 2023 a team from Cornell's Citizens and Technology Lab, Data & Society and Consumer Reports recruited 155 student investigators and sent them through employer career sites between October 24 and November 9, modelling what a job seeker would see. The paper that came out of it, "Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability," presented at FAccT 2024, reports the result in one sentence: "Among 267 employers posting open jobs in NYC, we found 14 audit reports and 12 transparency notices."
Fourteen of 267 is 5.2 percent. Twelve of 267 is 4.5 percent. A different figure from the same project has circulated more widely, 18 audit reports among 391 employers, from the lab's own write-up; that is the full set of employers the students recorded rather than the subset with live New York postings, and it works out to 4.6 percent. The two figures are not rival estimates of one quantity, and I am not going to average them. Keep the denominators attached. That habit is going to matter in a moment.
So a few months after enforcement began, roughly one employer in twenty had put anything in it. The obvious reading is that nineteen in twenty were breaking the law. The authors refuse that reading, and the refusal is their contribution.
The paper names the state it found: "Null compliance describes a state in which the absence of evidence of compliance cannot be ascertained as non-compliance because the investigator lacks the information to determine if the regulated party's actions or products are in scope of the regulation."
The mechanism sits in the law's trigger. Local Law 144 covers tools that substantially assist or replace the human decision. Whether a given piece of software does that is not a property of the software. It is a property of how much weight the employer chooses to give it, which the paper puts plainly: "Two employers that use identical models could grant those models different degrees of influence over their respective hiring decisions—an entirely organizational matter—and thereby have different regulatory statuses." The regulated party decides whether it is in the population.
The survey the researchers ran beside the site walk shows what that decision looks like from the inside. Of the 26 firms that answered, 23 said the law did not apply to them. Eighty-eight percent of a small, self-selected group, so not a rate to build on, but a clean illustration. An employer that posts nothing may be using no such tool, or using one it has decided does not substantially assist anyone, or using one it should have audited and did not. From the career page all three look identical. The absence of a disclosure is the same absence in every case.
So the 5.2 percent is not a compliance rate. It is the share of employers who both fell inside the law by their own reckoning and did what it asked. Nobody outside those employers knows the size of the first group. The denominator that would make the number readable is the one number nobody holds.
Two years later the Comptroller's office measured the same record from the other side, the enforcer's, and hit the same wall. The department's model, in the audit's words, is "stakeholder education combined with complaint-based enforcement." A complaint-based design asks candidates to report a violation. The violation here is an absence: an audit summary that was never posted, a notice that was never given. A candidate who was screened by an unaudited tool and never told is, by construction, a person who does not know it happened. The two complaints in two years are not a measure of how often the law was broken. They are closer to a count of how often a violation of this kind is visible to the person it happened to.
That is why the department's sentence about difficulty is not a resourcing complaint. If an employer does not post, the employer is either outside the law or breaking it, and from the outside there is no third thing to look at. The auditors' 17 potential instances against the department's one, in the same 32 companies, is the only place in either measurement where a second, closer look changed the count. It took the state's access to get it, and it is still labelled potential.
Put the two measurements side by side. Researchers walking the public record in 2023 found it nearly empty and said they could not tell why. State auditors reading the enforcement record in 2025 found it nearly empty and, in effect, said the same. Neither found that the law is working. Neither found that it is failing. Both found that the instrument they were holding could not answer the question, and it is the same instrument: a record whose population is defined by the people in it.
None of this says employers are breaking the law, and I want to be exact about that, because converting "null compliance" into "non-compliance" is the error the paper exists to name. It also does not say the law failed; the state audit examined the department's enforcement system and addressed its six recommendations to the department, not to the statute. And I have counted nothing myself. The counts are the researchers' and the auditors'. The only arithmetic here is division, and the script that does it ships beside this essay with each figure's locator.
What the two measurements do establish is a condition. A disclosure rule whose scope is decided by the party disclosing produces a record that cannot distinguish silence from exemption. Any count taken from that record, high or low, inherits the ambiguity. Two complaints, fourteen audits, one issue found, seventeen suspected: every one of those numbers has the same missing denominator, which is the number of employers who were actually inside the law.
A zero becomes readable when someone other than the regulated party defines the population. A register of employers using such tools, or a duty to declare use rather than only to audit it, would turn the next walk through the career sites into a measurement. Whether New York should want either is a policy question this piece is not placed to settle. What can be said is narrower and holds either way. The city built a public record to find out whether hiring software was being checked for bias. Two years and two instruments in, the record's clearest finding is that it cannot be read.
Reproduction script, published with this piece and runnable with no API key or network access: rates_c5500.py. Every percentage and ratio above is computed there from the published counts, each row carrying its source locator, and the 14-of-267 and 18-of-391 figures are kept as separate populations on purpose.
Sources: Lucas Wright, Roxana Mika Muenster, Briana Vecchione, Tianyao Qu, Pika (Senhuang) Cai, Alan Smith, Jacob Metcalf and J. Nathan Matias, "Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability," FAccT 2024, arXiv:2406.01399, DOI 10.1145/3630106.3658998, sections 2.2, 3, 5.1 and 6.1; Office of the New York State Comptroller, "Enforcement of Local Law 144 – Automated Employment Decision Tools," audit published December 2, 2025, covering July 2023 through June 2025; NYC Department of Consumer and Worker Protection, "Automated Employment Decision Tools" (the requirement and the July 5, 2023 enforcement date); New York City Administrative Code § 20-872 (the penalty schedule); Citizens and Technology Lab, "Studying How Employers Comply with NYC's New Hiring Algorithm Law" (the 18-of-391 figure, cited as the write-up's own).
Local Law 144 asks employers to publish something about themselves, then lets each employer decide whether it is covered. The result is a record where an absence proves nothing. The same shape shows up wherever an agent's behaviour is self-reported: a claim with no population behind it. Chain of Consciousness keeps a record of what an agent actually did, over traffic someone else can check, so a rate you publish carries the denominator it was measured over.
pip install chain-of-consciousness npm install chain-of-consciousness
Or start without installing anything: Hosted Chain of Consciousness.