Sumitsubo: Checking Implementations with Docs

This article is translated by AI, if have any corrections please let me know.

Sumitsubo is the tool I use to turn a “spec” into code that can actually be verified. In the early attempt covered by Sumitsubo: Turning Specs into a Linter I used JSON to describe the spec, but while looking into whether similar tools already existed, I unexpectedly found that making Markdown itself the structured source of information lets you design the tool by writing documents.

Structured Data

There were plenty of considerations behind picking JSON at first, including that LLMs (Large Language Models) are good at maintaining this kind of file format, and that mechanical processing needs structured data to work on. But I had forgotten that Markdown has those same properties.

For example, # and ## already express depth, and \n\n (two line breaks) expresses a paragraph of text. Essentially every part of it can be parsed, which is why turning Markdown into a web page is so easy. Better still, the syntax is tiny, so parsing it isn’t difficult.

So I spent another week moving the spec definitions from JSON over to Markdown, and removed the ability to render JSON into Markdown, because the two no longer need to coexist. Markdown alone expresses everything I originally wanted the spec to carry.

But a new difficulty showed up. Parsed Markdown all comes out in the same shape, so how do you express what different specs need?

Designing the Abstraction

Writing structured specs in Markdown turned out to be simpler than expected. It is really just the “abstraction” that software development does all the time. If we can turn a spec into some concept, then there should be a data type that describes every checkable spec in a uniform way.

While settling on that concept, I unexpectedly ran into ReqIf, a standard whose full name is Requirements Interchange Format. Put simply, it is a standard format for exchanging “requirements,” and those requirements happen to be the requirements of a PRD (Product Requirement Document), which can be seen as one type of spec document.

In other words, representing a spec as structured data is not a brand-new idea. It has just rarely been turned into a checking tool (Linter) before. Sumitsubo borrows the ReqIf concept and defines a spec as

• Specification - a spec file
• Statement - a single spec item, which can be nested

As for the nesting, # and ## are enough to tell the levels apart, and Markdown supports that design almost natively. Treating the heading as the identity is natural too, so a spec can be written like this.

1## `MD-001` Parse a Markdown Spec
2
3When the user provides a .md file that meets the conditions, parse it into a structured spec.
4
5| Given | .spec/glossary.md exists    |
6| When  | the user runs `sumi verify` |
7| Then  | it returns `exit 0`         |

Because it is so easy to write and maintain, and conveys the intent of the behavior well, it reduces the chance of the AI missing a requirement when implementing against the spec, and it stays quite maintainable for people too.

Comparing Syntax Trees

The move to Markdown was really driven by the need for a syntax tree. Expressing an interface contract through JSON has a hard time meeting the needs of different languages, which also made Tree Sitter’s multi-language support fairly pointless.

Fortunately tools like Semgrep and ast-grep already show what Tree Sitter can do. It can serve the purpose of “finding a piece of code that matches some condition,” and that property can be used to constrain the AI so that certain public methods stay stably defined during implementation, instead of changing all the time.

 1## `Sumitsubo::Place.of`
 2
 3Produces a Place object representing a location that some spec defines.
 4
 5```ruby
 6class Sumitsubo::Place
 7  def self.of(path, line)
 8  end
 9end
10```

Like this, I can require that the files within the search scope must contain a Place object with a class method called of(path, line). How it is implemented doesn’t matter, but I have to see that code exist.

When the AI one day breaks that agreement while making changes, sumi verify will show a message like this

1$ sumi verify
2.spec/contract/place.md:3 ruby Sumitsubo::Place.of is defined nowhere this specification includes, and one the reading cannot see never is
31 difference

telling the AI this is an agreement in the spec that shouldn’t be touched. Verifying an AI-generated implementation then narrows down to the scope of the document, rather than checking and searching at the level of the whole project.

A lot of the time what we care about is the public methods and the behavior. Private methods inside are encapsulated, so they aren’t necessarily the first thing to confirm. In an era where AI produces code in volume, checking gets much harder, but with a mechanism like this as a guard, the debugging scope shrinks and it becomes easier to communicate about.

Sumitsubo’s features are still at a very early stage, but I believe more tools of this kind will show up to meet the needs of future software development. Not just tests and syntax checking, but gradually aligning the spec (the intent) with the implementation. Even something like Sumitsubo, which can only check a small part, is still a bit better than not checking at all, or than having no way to check.