Department.Technology

Tag: Human Oversight

  • Build It With the Controllers: The Danger of Rolling Out SMART Without NATCA

    By the Founders, Department of Technology | San Diego, California | November 15, 2026

    The FAA is using MI (Machine Intelligence, aka AI or Artificial Intelligence) to help manage the nation’s airspace. The tool, called SMART, began limited operation around Washington, D.C. on September 21 and was expanded to all 22 air route traffic control centers on November 10, three days before the November 13 ground stop. The union that represents air traffic controllers says it had no part in designing, testing, or implementing it.

    We are not claiming that SMART is unsafe, or that it caused the ground stop. Officials say it did not, and the cause has not been confirmed. Our claim is narrower: when a critical system is rolled out without the people who use it every day, the risks go up, and the cost of avoiding them is small. This is the third post in our series. Our first post asked what accountable technology leadership would have changed, and our second asked what should be settled before $50 billion is spent.

    What we know so far

    Here is what has been reported. Most of the reporting on the union’s role dates from September.

    • The rollout. The FAA began limited use of SMART in the Washington area in September. It told FLYING that SMART was active in all 22 centers as of November 10, and AeroTime reported that the expansion came after lawmakers had sought a pause in the Washington-area rollout.
    • The union’s position. In September, the National Air Traffic Controllers Association (NATCA) said it had not been involved in the design, testing, or implementation of SMART and could not speak to its effect on the workforce, as NPR reported. FLYING reported that the union stressed that new technology should complement controllers’ judgment, not replace it.
    • A lawmaker’s account. Rep. Don Beyer called on the FAA to suspend SMART at Washington-area airports until controller feedback and independent stress testing showed it was safe. He said he had been told that controllers were not consulted on its design and development, were not trained on it before it went live, and had been told it can adjust schedules and potentially even routes, according to Common Dreams. That is his account, and we have not seen the FAA respond to it directly.
    • The FAA’s description. The FAA says SMART cannot replace controllers or take over an aircraft, and that every recommendation is reviewed by FAA staff, with local leadership free to accept or pass, per MeriTalk. The agency credits the Washington rollout with saving an average of 10 minutes per rerouted flight, according to AeroTime.
    • The ground stop. Officials said Friday’s outage was not related to SMART, as the Taos News reported, and we have no reason to doubt that.

    What we do not know

    Several basic questions are still open.

    • Whether the FAA consulted controllers or facility leaders in other ways. To our knowledge, the agency has not described what consultation took place.
    • Whether NATCA has been brought in since September. We found no union statement on the November expansion.
    • What testing was finished before the expansion, and what criteria had to be met first. Airways noted that the FAA’s November statement does not explain how its savings were calculated, give a comparison baseline, or report how often facilities accepted recommendations.
    • The cause of the ground stop, which has not been confirmed.

    Five dangers of skipping the people who use the system

    Everything in this section is our analysis, not reporting.

    1. A human review is only as good as the reviewer’s understanding

    The FAA stresses that staff review every SMART recommendation. That safeguard depends on people who understand what the tool is doing, trust it enough to use it, and doubt it enough to challenge it. People who were not involved in building or testing a tool, and who may not have been trained on it, are more likely to accept its output without question or to ignore it altogether. Either outcome defeats the purpose of keeping a human in the loop.

    2. Designers miss what operators know

    Controllers know how traffic really moves at their facilities, where the workarounds are, and what a recommendation looks like in the middle of a busy shift. That knowledge rarely makes it into a specification. Bringing it in during design is far cheaper than discovering its absence after go-live.

    3. Mixed messages about what the tool does

    The FAA describes SMART as advisory. According to Beyer, controllers were told it can adjust schedules and possibly routes. We cannot say which description is right, and the gap itself is the danger. In a safety-critical workplace, people need to know exactly what a tool can and cannot do, and who has the final say. A joint design and review process is how those answers get written down and shared.

    4. Speed without a shared record of what was learned

    About seven weeks passed between the first Washington-area use on September 21 and the nationwide expansion on November 10. A staged rollout is a sound idea, but a stage only teaches something if the results are examined, ideally by people with independent standing. As noted above, we have not seen public figures on how often facilities accepted SMART’s recommendations. Union representatives who help review each stage, against criteria agreed in advance, would give the process a credibility that an internal sign-off alone cannot.

    5. Goodwill is scarce, and so is public trust

    The FAA is still thousands of certified controllers short of full staffing, and many work mandatory overtime and grueling schedules, according to NPR. Asking a stretched workforce to adopt a major new tool it did not help shape spends goodwill the agency can ill afford. And when something goes wrong, as it did on November 13, a thin public record of how the tool was vetted leaves room for speculation. Headlines linked the ground stop to the SMART expansion the same day, even though officials said the two were unrelated. A visible partnership with the controllers’ union would have been part of the answer to that question.

    What investing the time would look like

    None of this requires slowing the work to a crawl. Based on the problems above, these are the steps we would suggest. They are our proposals, not the FAA’s plan.

    • An early invitation. Bring the union into design review before a system is built, and into testing before it goes live.
    • Training before go-live. Train every controller and traffic manager who will see the tool’s output, and give them a clear written statement of what it can and cannot do.
    • Joint review at each stage. Agree in advance on the criteria for moving from one stage to the next, and have the union help review the results.
    • A public report. Publish what was tested, what was found, and what changed, including how often recommendations were accepted or passed.

    Where reasonable people disagree

    Supporters of the current approach point to the staged rollout, the FAA’s reported savings, and the fact that SMART is advisory and reviewed by FAA staff. Airlines for America praised the push to modernize, according to NPR. Skeptics, including Beyer, argue that a tool with safety implications should be vetted with controller feedback and independent testing first. We think both sides have a point. Inviting the union in is not the same as giving it a veto, and it does not have to be slow. It is a way of finding problems early, while they are cheap to fix.

    Where a Department of Technology fits, and where it does not

    A Department of Technology would not design SMART, run air traffic control, or negotiate with NATCA. As described in our federal framework, it would audit major technology initiatives, require impact assessments, and report to Congress and the public. We would add one requirement that goes beyond the written framework: for any major system that changes how safety-critical workers do their jobs, the sponsoring agency must document how those workers and their representatives were consulted before rollout. That ties together two arguments we have already made: that changes should be tested before they go live, as in our first post, and that leaders should understand how systems work in practice, as in Back to Basics.

    Questions for officials and candidates

    1. Was NATCA invited to help design, test, or review SMART before the September launch, and before the November expansion? If not, why not?
    2. What testing and review had to be completed before SMART was expanded to all 22 centers, and who signed off?
    3. How were controllers trained on SMART, and what were they told it can and cannot do?
    4. How often do facilities accept or pass on SMART’s recommendations, and will that data be made public?
    5. Will you commit to involving the workforce and its representatives in design and testing for the next major system?

    The bottom line

    A tool that helps manage the nation’s airspace should be built with the people who keep planes apart, not delivered to them. The time and goodwill it takes to invite the union in are small compared with the cost of losing the trust of the workforce, or the public, in a system everyone depends on. Whether or not SMART had anything to do with November 13, the lesson is the same one running through this series: name who is responsible, test before you go live, and show your work.

    Read the rest of this series: The November 13 Ground Stop: What a Department of Technology Would Have Done Differently and Fiber First? The Questions to Ask Before $50 Billion for Air Traffic Control For the basics of our proposal, read What Is a Department of Technology? We will update this post as more is reported, including any response from the FAA or NATCA.

  • Self-Revising MI: Why Machine Intelligence That Rewrites Itself Needs a Plain-English Name

    By the Founders, Department of Technology | San Diego, California | November 14, 2026

    When a Machine Intelligence system can change its own code, its goals or its safeguards, what should we call it? MI (Machine Intelligence, aka AI or Artificial Intelligence) is the term we use throughout this site, and we explain why just below.

    The most common name for the self-changing kind is “recursive self-improvement,” and we think it is the wrong tool for the job. We propose a plainer one: Self-Revising MI.

    Recursive self-improvement AI or Self-Revising MI is and will be for the seeable future the most important, controversial, and civilizational altering aspect of MI in our lifetime.

    Whether you are for, against or undecided about SRMI (Self-Revising MI), a name matters more than it seems. Legislators, judges, regulators, reporters and voters all need to use the same words for the same thing. A good name has to be understood by a general reader on first hearing, and it has to hold up when it is written into a statute, a contract or an audit standard. Self-Revising MI meets both tests better than the alternatives we considered.

    A quick note: why we say MI, not AI

    We use MI, short for Machine Intelligence, instead of AI, short for Artificial Intelligence, because we believe it is the more accurate word. The intelligence these systems show is real, and it comes from a machine. “Artificial” suggests something fake or second-rate, which muddies how people think and talk about it.

    We also think the clearer word is better for people: for science, business, governance and law. It is the same idea behind this whole post. A plain, accurate name makes for clearer thinking and clearer rules, and we hope you will start using MI too.

    The problem with the current names

    “Recursive self-improvement” describes a specific, hypothesized scenario: a system improves itself, and the improved version is better at improving itself again, so progress could accelerate. That is an important idea, but it makes a poor category for rules. “Recursive” has a precise technical meaning in computer science (a procedure defined in terms of itself), and a system can change itself without any of that. “Improvement” assumes the change worked. A rule that only applies to systems that improve would force regulators to prove a success before they could act.

    Researchers use a crowd of overlapping labels. One 2026 survey of self-evolving agents sets its own boundary with three criteria: updates driven by experience, persistent effects, and self-initiated learning. A separate safety paper on self-evolving agent systems defines its scope with a different set of conditions.

    Earlier work on self-modification in rational agents uses yet another label. Each is useful inside its own field, but none gives a legislator or a judge a single, stable term.

    Our proposal, with a model definition

    Self-Revising MI is a Machine Intelligence system that is able to make persistent changes to its own code, model weights, training data or training process, objectives, tools or safeguards, without a person approving each individual change.

    Each part of that sentence is doing work.

    • “Self” means the system initiates or carries out the change, not an engineer.
    • “Revising” means a change to the system itself, not just to its output. Editing your own draft is the everyday picture.
    • “Persistent” excludes temporary adaptation, such as adjusting to the instructions in a single conversation. The 2026 survey above draws a similar line.
    • “Each individual change” means a one-time, blanket permission does not take a system out of the category. Approval has to be per change.

    This is an illustrative definition for discussion, not legal advice. Any real statute would need review by lawyers and technical experts.

    Four reasons “revising” is the right word

    • It describes a mechanism, not an outcome. A revision either happened or it did not. It can be logged, compared against the earlier version and reviewed. Oversight should not depend on arguing whether a change was an improvement.
    • It is plain English. Everyone knows what it means to revise something. No glossary is needed, which helps in a hearing room, a courtroom or a news story.
    • It is wide enough for how systems actually change. According to Arize, what changes in today’s self-improving agents is almost always context, prompts, retrieved examples or tool policy, while weight updates are rare. Meanwhile, the self-modification research notes that a system’s goals can change too. A definition that only covered retraining would miss most real cases. Naming the categories of change closes that gap.
    • It fits an oversight scale. One recent paper proposes grading systems by their revision authority, meaning the highest level at which a system can adapt itself without a human. That is the same idea we use below.

    Why not the other names

    • Self-Advancing MI assumes the change is progress. A rule tied to “advancing” invites a fight over whether any given change advanced anything.
    • Autogenic MI is precise but obscure, and the word already has other uses in other fields. It would need explaining every time.
    • Ouroboros MI is memorable, but it is a metaphor, and it sounds ominous. Metaphors do not make good legal categories, though it may survive as a nickname.
    • Bootstrap MI borrows from computing, where “bootstrap” already means several different things, including starting up a system and a statistical method.
    • Compounding MI describes a trajectory, not a capability. A system can revise itself without its gains ever compounding.
    • Self-Directed MI sounds like it is about goals and autonomy, and could be confused with systems that choose their own tasks but never change themselves.

    Keep “recursive self-improvement” for the scenario

    We are not asking anyone to retire the older phrase. “Recursive self-improvement” (or an “intelligence explosion”) can stay as the name of the runaway scenario that researchers worry about. Self-Revising MI is the observable, regulable capability that could lead there. Rules should attach to the capability we can see today, not to a scenario we are still debating.

    A three-level scale

    A clear name works best when it is part of a ladder. Here is a draft scale for discussion.

    1. Fixed MI. Changes only when people retrain or update it. Standard testing and change logs apply.
    2. Human-Approved MI. May propose changes to itself, but a named person approves each one. Proposals and approvals are logged.
    3. Self-Revising MI. Can change itself without per-change approval. Stronger safeguards apply: independent audits, tamper-resistant change logs, tested rollback, staged deployment and public reporting.

    This also matches how the technology is described today. Arize says the common shipping pattern is that automation detects problems and proposes fixes while a person approves them, which is Level 2 on our scale. And EM360Tech reports that OpenAI said in September 2026 that fully autonomous recursive self-improvement is not happening today, and that Anthropic says it is not there yet either, though both report that their systems are doing more of the work of building future systems. If that is accurate, the time to agree on vocabulary and ladders is before Level 3 becomes common, not after.

    What this means for a Department of Technology

    A Department of Technology would not decide how any individual system is built. Its job would be the accountable layer around it: keeping a shared vocabulary, requiring change logs, auditing the highest-risk systems and publishing what it finds. The same logic applied to the November 13 ground stop in our earlier post: when a critical system changes, someone should be able to say what changed, who approved it, and how to undo it.

    What we are asking

    • Use “Self-Revising MI” when you mean Machine Intelligence that can change itself without per-change human approval.
    • Ask officials and candidates to adopt a defined term, with categories of change named in the definition.
    • Ask developers and agencies to log every change a system makes to itself, and to say which level each system operates at.

    The bottom line

    The best names say what happened, not what we hope or fear will follow. Self-Revising MI tells you that a system changed itself, and that is the fact oversight needs first. It is clear to a general reader, specific enough for a statute, and leaves room for the older terms to keep their place. We will keep refining the definition, and we welcome your comments.