Sumitsubo: Turning Specs into a Linter

contact@aotoki.me (Aotokitsuruya) — Wed, 26 Aug 2026 00:00:00 +0800

Midway through building Kobako, the question of whether the spec and the implementation were still aligned kept bothering me. Spec-Driven Development is ideal in theory, but language models are not deterministic by design, and even with a spec to refer to they still drift, or leave things out.

So Sumitsubo came together recently as a general-purpose checking tool.

The Ideal and the Reality

Proving that a spec has actually been implemented is hard to confirm efficiently unless you run the thing and read the source yourself, and it only gets harder once the project grows and becomes more complex.

If we want work that resists mechanization to have a chance of being handled mechanically, structured data is a good option. Anthropic’s Effective harnesses for long-running agents last year happened to show one promising possibility, registering the spec as JSON and then letting Claude Code implement it.

In practice this still cannot guarantee perfect coverage, but it at least proves that maintaining JSON is no longer difficult for AI. Maintaining features according to what the JSON says is reasonably stable, and it runs smoothly enough for most of the features to get implemented.

Ideally there would be some magical way to map a piece of natural language onto a piece of implementation, but that is hard to pull off in reality. Taking a step back and using a “claim” instead, letting the AI assert that a given spec has been implemented, is an acceptable consideration in the trade-off between checking nothing at all and checking perfectly.

Guiding the Agent

Before Sumitsubo existed, Kobako checked things with an “anchor” mechanism. When the spec defined behavior B-25, that B-25 had to appear at least once in a test comment to pass, which to some extent nudged the AI into working out why a feature the spec mentioned had no test covering it.

Moving to Sumitsubo, the approach became a “behavior marking” implemented in a BDD (Behavior-Driven Development) style paired with an identifier. This is not a new idea, Gherkin has its own @B-25 tagging mechanism, but I turned the manual maintenance into something the AI handles.

 1{
"name": "Behavior",
"description": "The scenarios a project declares, and where each one sits.",
"include": [
"test/behavior_test.rb"
],
"scenarios": [
{
"id": "B-001",
"title": "What the directory declares, and where",
"given": [
"a behavior directory holding more than one specification"
],
"when": "the directory is loaded",
"then": "every scenario answers with the line of the file that declares it"
}
]
18}

Above is one of the behavior rules Sumitsubo maintains for itself, kept under the .spec/behavior/ directory. B-001 confirms that every spec gets loaded, so in behavior_test.rb you can find a marking like this.

 1# @behavior B-001
 2puts "--- what the directory declares, and where ---"
 3Sumitsubo::Behavior.load("test/fixtures/behavior/.spec/behavior").each do |feature|
 4 puts "#{feature.name} #{feature.includes.inspect}"
 5 feature.scenarios.each do |scenario|
 6 puts " #{scenario.path}:#{scenario.line} #{scenario.id} #{scenario.title}"
 7 scenario.given.each { |state| puts " given #{state}" }
 8 puts " when #{scenario.action}"
 9 puts " then #{scenario.outcome}"
10 end
11end

This way the AI shows for itself that “I have tested this rule” rather than saying so with nothing to back it up. Without the behavior marking, sumi verify reports a message like this

1$ sumi verify
2.spec/behavior/behavior.json:6 @behavior B-001 is claimed nowhere in test/*_test.rb
31 differences

During implementation the AI then notices the missing piece on its own and goes on to add a test for it, which gives some degree of confidence that a given spec has been tested at least once.

In practice the marking only says the test claims to cover that spec, it does not guarantee the test really verifies it. This is still a claim rather than a proof, but compared with having no correspondence at all, the gap at least becomes visible.

The test above looks the way it does because it uses Spinel, the new project Matz announced at RubyKaigi in Japan this year, which compiles Ruby into C ahead of time and then turns it into a static executable. The test therefore runs the real executable and compares its output instead of verifying with assertions

Specs That Maintain Themselves

Building Sumitsubo made me wonder whether Spec-Driven Development is still necessary, or whether it will really fade away as Harness Engineering matures, because I noticed something quite interesting once Sumitsubo was used to check itself.

The reason we moved toward spec first and implementation second was that code alone is too hard to understand, and in past software development experience documentation and implementation were very hard to keep aligned. In the AI era that alignment is achievable to some degree, so spec first really did become a sensible mechanism, especially when AI was weaker and the guidance mattered.

While building Sumitsubo, though, I wanted it to maintain its own spec using itself, so I deliberately did not start from SPEC.md (my preferred spec document) and went from scratch instead. Once the behavior marking mechanism was set up on the project together with a Claude Code Hook, the AI was able to spot spec discrepancies and correct them on its own.

For example, Sumitsubo’s “marking” mechanism records a rule that a comment with no code after it does not count as a marking. When I started adding Rust support for Kobako and abstracted out the concept of Language, the AI noticed on its own that this rule sat more sensibly with the language behavior than the marking behavior, and went ahead and adjusted the markings, revised the behaviors, and so on.

When the tooling is in place, designed properly that is, perhaps many of the mechanisms that rely on prompts are what actually constrain the AI. Suppose we hand it a systematic tool for maintaining specs, then an instruction as simple as “use sumi verify to check the spec” might be enough to keep most of the spec in good shape.

I am a fairly firm believer in Spec-Driven Development, yet once I turned the spec into something verifiable, a linter, I genuinely started to doubt it. I am still not sure how this will develop, but it is certainly a new direction worth trying.

SDD on Aotokitsuruya is learning

Sumitsubo: Turning Specs into a Linter

The Ideal and the Reality

Guiding the Agent

Specs That Maintain Themselves