AI proof automation is getting good enough that product teams should pay attention. It does not remove the need for product boundaries around generated software.
Adam Langley published a useful example this week. He used LLMs while building a Zstandard decompressor in Lean and argued that LLMs may make dependent-type systems far more practical as proof automation.1 The interesting part was not a promise that models can write all correct software. The interesting part was narrower: if a formal statement is right, the proof can often be produced by a machine.
That matters for AI app builders inside SaaS products. These systems create dashboards, forms, reports, and small workflows from a prompt. Some of those apps only read data. Some eventually write fields, start approvals, or change a customer workflow. When generated software gets that close to the product, "the model seems smart" is not a safety model.
Key Takeaways
- The seL4 team reported about 10 times as much proof effort as design and implementation work, plus more than 20 times as many lines of proof code as C code.1
- LLMs can reduce the proof-writing burden, but they do not decide which product actions are safe.
- AI app builders still need narrow API scopes, review states, audit logs, and rollback around generated apps.
What is AI proof automation? #
AI proof automation is using a model to help produce formal proofs that software satisfies a stated property.
In a language such as Lean, the developer can write a claim about how a function should behave. The proof checker verifies whether the proof is valid. The model can help fill in the proof steps, but the checker is still the authority. That distinction matters. The model is not trusted because it sounds confident. The proof is trusted because another system checks it.
Langley's example is a good fit for this pattern. A decompressor has a tight technical job. It has a specification. You can ask whether an output is valid for an input. That is a much better target for automation than an open-ended product question such as "should this customer workflow be allowed to update records?"
Why is this suddenly relevant to AI app builders? #
AI app builders are moving from prototypes toward product surfaces that real users share.
A generated dashboard is easy to understand. A generated form that changes state in the host product is harder. A generated workflow that reads from one API, writes to another API, and sends a notification is harder still. The more useful the generated app gets, the more it needs product rules around it.
On sales calls, technical buyers tend to ask practical questions before they ask about model benchmarks. Where does the generated code sit? Which APIs can it call? Can it write back to the database? What happens before a customer shares it? Those are the right questions because they are asking where the boundary lives.
Proof automation can help with part of that answer. It may become easier to check that a generated app calls a function with the right arguments, handles a required field, or preserves a small invariant. But the product team still has to decide which invariants matter.
What can formal checks actually prove? #
Formal checks are best at narrow claims with clear inputs and outputs.
For an AI app builder, useful checks might include:
- this app only calls approved API routes;
- this field is never sent to the client;
- this workflow cannot write outside its tenant;
- this generated form cannot submit without required fields;
- this app handles empty states without throwing;
- this publish step creates a version that can be rolled back.
Those are valuable claims. They are also smaller than "this app is good" or "this app is safe." A proof checker can verify a property that was stated precisely. It cannot know whether a customer success team should be allowed to create a new approval workflow for one enterprise account.
That is why product boundaries matter. The safest generated app is not the one with the longest prompt. It is the one running inside a small box with clear permissions.
Where do prompts stop helping? #
Prompts stop helping when they try to replace enforceable product rules.
A prompt can say, "do not expose sensitive data." The runtime still needs field-level access checks. A prompt can say, "only call safe APIs." The tool layer still needs an allowlist. A prompt can say, "ask for review before publishing." The product still needs a publish state that prevents the app from spreading until someone approves it.
This is the same reason proof automation is promising. We do not want to rely on the model's self-report. We want a separate check. The same logic applies to AI-generated SaaS workflows. The model can draft the app. The product has to constrain, inspect, approve, and log what the app does.
Gigacatalyst is built around that split. The AI generates focused apps inside a SaaS product, but those apps still run through the host product's APIs, permissions, design system, review flow, versioning, and audit trail. The model gets enough freedom to be useful, not enough freedom to invent a new security boundary every time a user types a prompt.
How should SaaS teams pilot this safely? #
Start with one workflow where the allowed actions are obvious.
A good first test is usually a read-heavy workflow: a prospect-specific demo dashboard, an account health report, or a customer journey view assembled from approved fields. Give the builder a small API set. Define the user role. Decide whether the result can be shared or only reviewed. Then inspect the generated app as a product artifact, not a chat transcript.
For write paths, start even smaller. Pick one table, one allowed field group, one role, and one review step. If the generated app needs more power, add it after the team sees the actual failure mode. That is slower than saying "the model can do anything," but it is much faster than cleaning up an app that did the wrong thing across customer accounts.
Want generated apps with product boundaries?
Gigacatalyst helps B2B SaaS teams let customers and internal teams build focused apps on approved APIs, with permissions, review, versioning, and audit around the generated output.
FAQ #
Conclusion #
Proof automation is a reason to be more ambitious about generated software, not more careless with it.
The useful pattern is simple. Let the model generate the app. Let product boundaries decide what the app is allowed to touch. Let separate checks verify the parts that can be stated precisely. That is how AI-built software becomes something a SaaS team can ship, not just something a model can demo.
Sources #
Footnotes #
-
Adam Langley. "We have proof automation now." ImperialViolet. https://www.imperialviolet.org/2026/07/26/zstd-lean.html. 2026. ↩ ↩2