Steno & PL

🙏: please or thank you?

Fri, 03 Apr 2026 00:00:00 +0000

Intended audience	Linguistics enthusiasts. Autistic individuals working in a professional setting with access to emoji.
Occasion	April Cools' 2026 — late as usual!
Origin	Original research from 2025-02-28 Informal surveys from a few of my group chats.
Mood	Curious.

Table of contents:

Question
Results
Responses
Related posts
Comments

At work, I started using the 🙏 emoji to reply to Slack messages. Then I realized that it might have different meanings about

either the past: expressing thanks, appreciation, or sympathy,
or the future: expressing a hope, wish, or request,

and maybe not everyone uses it the same way that I do.

It would be unfortunate if someone thought I was expressing gratitude for their situation when I meant to express sympathy!

Question

I surveyed various of my group chats with the following poll:

Which of the following situations do you use 🙏 emoji for?

Indicate request (“please”)

Indicate gratitude (“thanks”)

Indicate hope for a future event

Indicate sympathy or commiseration for a past event

Other (add your own option)

Results

Situation	A1	A2	A3	B1	B2	B3	B4	C1	C2	D1	Total
Indicate request (“please”)	✅	✅	❌	✅	✅	✅	❌	✅	✅	✅	8
Indicate gratitude (“thanks”)	✅	✅	✅	✅	❌	✅	❌	✅	✅	✅	8
Indicate hope for a future event	✅	✅	✅	❌	✅	❌	✅	✅	✅	✅	8
Indicate sympathy or commiseration for a past event	❌	✅	❌	✅	✅	❌	❌	❌	❌	❌	3

Some take-aways:

More commonly: request/gratitude/hope were equally (!) common.
Less commonly: some people use it to indicate sympathy/commiseration for a past event, but one person even expressed explicit confusion about this meaning.
Less commonly: it can have connotations of prayer. (Note that the official description is PERSON WITH FOLDED HANDS, which doesn’t specifically indicate prayer.)

This suggests that my original concern of expressing gratitude instead of sympathy is not unjustified.

Responses

Chat A (n = 3)

Indicate request (“please”) — A1, A2
Indicate gratitude (“thanks”) — A1, A2, A3
Indicate hope for a future event — A1, A2, A3
Indicate sympathy or commiseration for a past event — A2

A1:

[…] I use it for the first three

A2:

I don’t use it but if I did those all seem like situations where it’s applicable

A3:

Sometimes thanks (mostly I see it used by others and I don’t find it cringe)

If you mean it as please, it might come off as over-emphatic/passive aggressive

Hope for a future event (like “I pray”) - in a somewhat humorous context of a group chat it is entirely appropriate

I don’t get how it fits the sympathy/commiseration context

Since it has a praying connotation and because it also looks like a gesture used more by some cultures than others, I try to avoid it myself for fear of seeming insensitive/being misunderstood

Chat B (n = 4)

Indicate request (“please”) — B1, B2, B3.
Indicate gratitude (“thanks”) — B1, B3.
Indicate hope for a future event — B2, B4.
Indicate sympathy or commiseration for a past event — B1, B2
➕ Replacement for “care” emoji where there isn’t one — B2
➕ Praying — B4.

B3:

I’d use 🤞 for hope

Chat C (n = 2)

Indicate request (“please”) — C1, C2
Indicate gratitude (“thanks”) — C1, C2
Indicate hope for a future event — C1, C2
Indicate sympathy or commiseration for a past event — (none)

Chat D (n = 2)

Indicate request (“please”) — D1
Indicate gratitude (“thanks”) — D1
Indicate hope for a future event — D1, D2
Indicate sympathy or commiseration for a past event — (none)

D2:

I use it as „praying for you” […] Like if someone is having trouble with something

The following are hand-curated posts which you might find interesting.

Date		Title
24 Aug 2016		Steno Journal: Weeks 1-12
06 Sep 2016		Stenography adventures with Plover and the Ergodox EZ, part 1
06 Sep 2016		Stenography adventures with Plover and the Ergodox EZ, part 2
13 Oct 2016		Analyzing all of my Messenger conversations to create conversational macros
21 May 2018		My steno system
02 Jan 2023		On bullet points
11 Jan 2023		Interactive blogs
05 Apr 2025		Language-learning anecdotes
03 Apr 2026	(this post)	🙏: please or thank you?

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Thu, 30 Oct 2025 00:00:00 +0000

Intended audience	Software engineers interested in logic programming or static analysis, and actually using it in production.
Origin	Originally presented at Seattle Build Enthusiasts Meetup (Autumn 2025) Frustration at work due to not having a hermetic build system. Prior experience with static analysis techniques.
Mood	Excited to share a useful tool with like-minded colleagues.

Table of contents:

Slides
Speaker notes
Related posts
Comments

Slides

This talk was given at the Seattle Build Enthusiasts Autumn meetup (2025-10-30).

Main goal: Show some use-cases where Datalog shines, and try to convince the audience to use it.

Resources:

Direct Google Slides link
Ascent homepage
I’ve included rough speaker notes below, and also transcribed a few of the interesting Ascent code snippets directly inline.

Speaker notes

(1) Hi everyone, my name is Waleed Khan. I work at my company not on the Bazel team. Therefore, I’m going to show you how not to migrate Bazel.

Instead, I’m going to try to sell you on a cool tool, which is a library for Datalog, to rapidly build certain kinds of analyses, especially graph-oriented ones. And I’ll show you how I used it in our repository to statically detect entire classes of misconfigured CI jobs, and fix them before they broke the build.

The context for this is that, recently, we had a “Quality Week” at work. I took the initiative to enable a few rules for ESLint in our monorepo. (If you’re not familiar, ESLint is a linter for JavaScript and Typescript).

I just turned on the rules in the global config file, opened a pull request, and saw a bunch of new lint violations in CI, as expected. Then I either fixed or suppressed each of them, got CI green, and finally committed my change.

Naturally, this immediately broke a bunch of builds. That’s because our CI tries to run only relevant jobs, and, in this case, it was wrong about which ones were “relevant”.

(2) How could this possibly happen? Well, here’s what our CI job definitions look like. There’s a set of globs that match changed files, and they trigger the corresponding build and test commands.

As Build System Enthusiasts, you’ll probably spot two problems here.

The first one is that this is a hellish custom YAML-based configuration language, which is its own entirely separate issue.

The second is that there is no guarantee that this set of globs is necessary or sufficient to trigger the associated command. Notably, if my/package has a dependency on some other directory in the codebase, then changes to that code won’t automatically cause this job to trigger.

(3) So what can we do about it?

The best option is probably to use a hermetic build system like Bazel. Then this kind of stuff just can’t happen.

And I specifically asked Dan if he could please spend a week or two on it, and just quickly migrate the entire codebase to Bazel, but I guess he’s busy with something else, because it kind of looks like the codebase has not yet been migrated to Bazel? I’m sure he’ll get around to it.

So what can I do in the meantime to kind of stop the bleeding, as a kind of band-aid solution?

Well, If we look back at this job definition, we could probably make a guess about the necessary dependencies here. Like, if the script is called “lint”, then it probably depends on linting configuration files.

And you know what’s really good at making unsubstantiated guesses? That’s right. AI. But I didn’t want to wade through a bunch of noisy, non-deterministic AI output to flag the real issues.

So, instead, I turned to a different approach, which is: AI. By which I mean “abstract interpretation”. We’ll discuss exactly what that means, but it’s basically a form of static analysis.

(4) Diagram: https://excalidraw.com/#json=DVyPsfKEa1RYdy62i5DW9,dVPy7Cm8QLVeEyaUcNuCwQ

So I wrote a static analysis tool using Datalog, and I’ll just show you an example of an actual trace that the tool managed to produce.

In this case, the CI job calls the npm script called “test” inside the my/package directory. And if we look at the package.json file in that directory, we can see that “test” is defined to call some tools.

But, misleadingly, it’s not just running tests. It’s also calling “npm run lint”. And if we go look at the “lint” script, then we see, “oh, it’s calling eslint in this directory”.

Now I know from the documentation that eslint always looks for configuration files named in certain patterns. And if we look for the associated configuration file, we can follow the imports and see that it extends from a different configuration file.

And that configuration file is not matched by the job’s globs. Notably, it’s not even the configuration file in the root of the monorepo, and I don’t even know why that was the case, but it is what it is.

The point is that, by a series of simple logical steps, we can automatically conclude that the globs must be missing a pattern to match this parent configuration file.

(5) So I looked at this and asked myself, “how can I quickly prototype a cross-language analysis that can detect these kinds of issues?” I decided to use Datalog for this.

Has anyone used Datalog or Prolog before?

Datalog is basically like: what if SQL didn’t suck? Datalog uses the relation model, like SQL. But it’s much better than SQL, and many standard programming languages, at expressing specific kinds of problems.

For example, this is how you express transitive closure in Datalog. There’s two “relations”, which are kinds of “facts” that the Datalog engine infers as it runs. In this case, we’re provided a “ground” set of “edges”. And we infer that a node is reachable from another node via some sequence of edges when either: as the base case, there’s a direct edge between them, or, as the recursive case, if there’s a transitive sequence of edges between them. That is, if x can reach y, and y is a neighbor of z, then x can reach z.

The syntax is a little backwards because the conclusion comes before the premises, but you get used to it.

You can see that the SQL version of this is pretty awful to write in comparison.

(6) So what kinds of things is Datalog good at? I have a few things listed here.

Recursive algorithms. That also includes situations where there might be “non-determinism”, in the sense that the algorithm might choose one of many possible futures. Datalog engines can also parallelize this stuff pretty easily. Graph analysis, including dependency graphs and transitive closures. Static analysis, including control flow and data flow analysis. “Rules” engines. When you want to express a bunch of “rules” succinctly and declaratively, and not focus so much on the implementation. Datalog is especially good if you find yourself needing to have multi-directional indexes or “joins”, or if you have rules from several domains that you want to combine, which is what my analysis did.

(7) I would typically not want to use a domain-specific language unless it granted me an order of magnitude more expressiveness. But Datalog delivers. Here’s an example from a blog post that one of my colleagues wrote, where he was working on some compiler analyses, and I challenged him to write it in Datalog instead.

So we did, and you can see the result here; the stuff on the left translates to two statements of Datalog. And, actually, that was the cleaned up version, because, originally, we only had one statement that we later broke into two statements for readability.

(8) So how do we actually use Datalog? I used a library called “Ascent”, which is a Datalog DSL for Rust. The fact that the tool is just a library is pretty important because it means that you don’t have to introduce some weird academic binary into your toolchain.

It’s also important because you can still just drop into Rust for various things. Like if you want to just use a Vec or a HashMap, you don’t have to fight with the language to get things done.

You can also use this library to compile to WASM, which is useful if you need to deploy it to other platforms.

(9) Here’s an excerpt from the original paper, where they implement the core of the Rust borrow checker in 46 lines of code. I, personally, couldn’t describe the Rust borrow checker in 46 lines of prose.

(10) So here’s a full example. You can see that we populate the initial “edges” relation with this expression using the “edges” parameter. The ascent_run macro will then run the program and return all of the relations, and we just access the “reachable” relation and convert it into a sorted set and return it.

(11) So, given these edges, the output would look like this. We can see that there’s an entry for (0, 3) because it’s reachable by visiting all the edges.

(12) Another great feature is that you can parallelize your analysis pretty easily by just adding “par” to your invocation. When you structure your analysis as a Datalog program, it also becomes pretty easy to parallelize it.

(13) Here’s a simple example which finds any local development services that run Bazel binaries, but which aren’t tested in CI. The “tdep” here stands for “transitive dependency.

Code listing:

// Check: Bazel targets referenced by services that aren't tested in CI.
relation service_untested_target(String, BazelLabel);
service_untested_target(service, label) <--
    dep(?Node::Service { name: service }, ?target @ Node::BazelTarget { label }),
    !is_ci_job_tdep(target);

(14) GitHub issue: https://github.com/bazelbuild/bazel/issues/27496

It turns out that, when using bzlmod, there’s no way to determine that a go_library target depends on the repo’s go.mod. In that case, we can argument the dependency graph to add this dependency. Note that we’re modifying the existing dep relation to add new inferred dependencies.

Code listing:

// Rule: All `go_*` targets depend on `go.mod`. (Actually, there should be
// more dependencies; I don't see how to query the bzlmod graph for them, so
// just hardcode this rule for now.)
dep(target, Node::BazelSourceFile { path: PathBuf::from("go.mod") }) <--
    bazel_target_is_kind(?target @ Node::BazelTarget { label }, kind),
    if kind.starts_with("go_");

(15) (remark on how we can express the semantics of Bazel ellipsis (/...) in just two fairly-straightforward rules: one to pretend that ... is a target that depends on its siblings, and one to pretend that subpackages are a type of target)

Code listing:

// Rule: Bazel label '//pkg:...' depends on all sibling '//pkg: target's.
// Note that sibling subpackages are represented as sibling targets, so this
// recursively includes all subpackage targets.
dep (
    Node::BazelTarget(repo.clone(), package.clone(), "...".to_string()),
    Node::BazelTarget(repo.clone(), package.clone(), target.clone()),
<——
    dep(?Node::BazelTarget(repo, package, target), _),
    if label.target != "...";

// Rule: For every package (represented as a target named '//:pkg of
// made-up type 'package), create a synthetic target named '//pkg:...'.
dep (
  Node::BazelTarget (repo.clone(), package.clone(), subpackage.clone()),
  Node::BazelTarget (repo.clone(), format! ("{package}/{subpackage}"), "...".to_string()),
) <--
    dep(
        ?Node::BazelTarget(repo, package, subpackage),
        Node::BazelTargetKind("package". to_string()),
    );

(16) So getting into the meat of it, here’s an actual rule. This detects the general case of “CI jobs that call Bazel somewhere, but whose globs don’t include all of the Bazel dependencies”.

It’s a lattice instead of a relation for technical reasons. In this case, it helps us produce the shortest trace, instead of all traces.

So, if we inferred any transitive dependency path between a CI job, and a file that we know is a source file for a Bazel target, via this trace,
and if we successfully processed that CI job, because I didn’t implement everything, so I just left some of them as “unimplemented”,
and if we look up the globs for this CI job, and none of them match that source file’s path,
then we fetch the glob matcher object, which basically just contains the list of globs so that I can present it to the user later,
and we fetch the source file that defines the CI job, which is expressed as a dependency too, because that information was actually resolved in another rule,
then we record the result for this job and source file pair (and you can ignore the fact that I store both the node and the path here, that was just for convenience).

Code listing:

lattice bazel_glob_miss(String, Node, PathBuf, ascent::Dual<GlobMiss>);
bazel_glob_miss(job, node.clone(), path.clone(), ascent::Dual(glob_miss)) <--
    ci_tdep(
        ?ci_job @ Node::CiJob { job },
        ?node @ Node::BazelSourceFile { path },
        ?ascent::Dual(trace),
    ),
    !ci_job_has_error(job),
    !ci_job_glob_matches_path(ci_job.clone(), path.clone()),
    ci_job_glob_matcher(ci_job.clone(), glob_matcher),
    dep(ci_job, ?Node::SourcePath { path: source }),
    let glob_miss = GlobMiss {
        job: job.clone(),
        source: source.clone(),
        path: path.clone(),
        trace: trace.clone(),
        glob_matcher: Arc::clone(glob_matcher),
    };

(17) (remark that “facts” are just a big enum which I serialize/deserialize with serde)

(18) (explain how I just dumped the Bazel dependency graph into a list of “facts”)

(19) (show how we can handle these 9 data sources in the same system and track cross-language control-/data-flow; remark that strace is different from all the rest, since it’s dynamic instrumentation, but it works compositionally anyways)

(20) Diagram: https://excalidraw.com/#json=DVyPsfKEa1RYdy62i5DW9,dVPy7Cm8QLVeEyaUcNuCwQ

(remark on how the original trace crossed between YAML, shell, package.json, and Javascript languages compositionally)

(21) (brief explanation of abstract interpretation)

(22) (explanation of simple shell abstract interpretation:

simple abstract interpretation: split on statement separators and pretend all commands run in sequence; not correct, but works in practice
track calls to cd and set internal state
track calls to bazel, npm, etc.

)

(23) (snippet of CESK machine from the Ascent paper, and remark that you can basically copy the rules, Greek letters and all)

(24) (links to resources)

The following are hand-curated posts which you might find interesting.

Date		Title
09 Apr 2017		How to parse contextual keywords in a programming language
24 Jul 2017		Null-tracking, or the difference between union and sum types
23 Dec 2017		Why LINQ syntax differs from SQL, list comprehensions, etc.
20 Apr 2020		Monotonicity is a halfway point between mutability and immutability
14 Jun 2022		Functions are constants too
19 Oct 2022		Build-aware sparse checkouts
17 Jun 2023		Encoding ML-style modules in Rust
24 Aug 2023		Writing brittle code
01 Sep 2023		On trivial changes
30 Dec 2023		Testing terminal user interface apps
02 Oct 2024		Incremental processing with Watchman — don't daemonize that build!
14 Aug 2025		Lithe, less analysis with Datalog
23 Aug 2025		Half my work is adding a cache
30 Oct 2025	(this post)	Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Half my work is adding a cache

Sat, 23 Aug 2025 00:00:00 +0000

Intended audience	Back-end software engineers, particularly in big tech.
Origin	Experience in big tech working on highly-scalable developer tooling. Sophie Alpert's blog post Materialized views are obviously useful.
Mood	Frustrated. Determined that we should do better as an industry.

Table of contents:

Related posts
Comments

When I was a kid, I wrote a little social networking site for my school. Back then, in 2010, I was surprised to learn that MySQL didn’t seem to support “materialized views”. Of course, I didn’t even know the name for the concept back then; but I understood that the concept should exist.

In her blog post Materialized views are obviously useful, Sophie Alpert points out that incremental view maintenance is still unreasonably difficult for trivial applications. And it’s 2025. In those 15 intervening years, we’ve barely accomplished anything as an industry to make incremental computation more tractable for the masses.

At best, we have DBSP: Automatic Incremental View Maintenance for Rich Query Languages (2022), which is a great paper and formalization, but was also probably realizable in 2010. We have a few startups (Materialize, Feldera) who offer incremental view maintenance as a service. More expansive efforts like Skiplang petered out. This is sad.

I’ve reproduced my Lobste.rs comment on the aforementioned blog post below.

This has been my thesis for a while:

Half of my work in big tech has just been “adding cache” or “removing a cache” in response to scaling latency/throughput requirements.
We absolutely need higher-level primitives for developing incremental systems, particularly in distributed contexts.
Somehow, our best alternative at each point has been to just build another ad-hoc, informally specified, bug-ridden, slow implementation of a build system.

I’m excited for efforts like Feldera (DBSP) to help provide

concretely, a specific implementation of a “build system as a library”, but also
abstractly, a reliable pattern that can be reimplemented across multiple environments where it’s unavailable (kind of like ReactiveX).

I think the main missing piece is that it’s hard to reconcile “persistent state” across many different environments/runtimes in a well-patterned way:

DBSP gives you a formula for effective incremental computation, but I don’t yet know how hard it is to wire it up to arbitrary data sources for input and output.
More specifically, the pattern seems to require that the underlying data source has a linear notion of “time”, which is not how most people have been designing their databases in practice.
- I can’t go take an arbitrary database at work that somebody implemented five years ago and easily ask Postgres for the delta stream.
- I mean, I’m sure it’s possible, but if it’s not dead simple, then I as a random developer am not going to be equipped to do it.

The following are hand-curated posts which you might find interesting.

Date		Title
20 Apr 2020		Monotonicity is a halfway point between mutability and immutability
19 Oct 2022		Build-aware sparse checkouts
02 Oct 2024		Incremental processing with Watchman — don't daemonize that build!
14 Aug 2025		Lithe, less analysis with Datalog
23 Aug 2025	(this post)	Half my work is adding a cache
30 Oct 2025		Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Lithe, less analysis with Datalog

Thu, 14 Aug 2025 00:00:00 +0000

Intended audience	Static analysis enthusiasts Logic programming enthusiasts
Origin	Recent experience with Soufflé Datalog. Discussion with Max Bernstein about his blog post Linear scan register allocation on SSA, resulting in his blog post Liveness analysis with Datalog.
Mood	Liberated

Table of contents:

Soufflé
Rapid prototyping
The curse of brevity
Related posts
Comments

Soufflé

I recently experimented with Soufflé Datalog for a certain work problem: comparing CI jobs’ declared dependencies with dependencies inferred via abstract interpretation (which is worth a separate post).

Recently, I saw the blog post Linear scan register allocation on SSA and noted that there was quite a lot of imperative implementation code, to the point where it might obstruct experimentating with the underlying ideas.

I suggested that the author could use Datalog instead to simplify the analyses, and we worked through the example of rewriting the liveness analysis in Soufflé, resulting in the blog post Liveness analysis with Datalog.

Rapid prototyping

As per its homepage, Soufflé promises to enable “rapid prototyping”:

Rapid-prototyping for your analysis problems with logic; enabling deep design-space explorations; designed for large-scale static analysis; e.g., points-to analysis for Java, taint-analysis, security checks.

And from my recent experience, I found that it delivered!

For example: When I wanted to implement the rule that, in Bazel, the target foo/... expands to all targets under foo, I just had to add one rule:

BazelLDep(repo, package, "...", $BazelTarget(repo, package, target)) :-
  BazelLDep(repo, package, target, _),
  target != "...".

It can be read as:

A target of the form <package>/... has a dependency on every target of the form <package>/<target>, as long as that target is not ....

And then this rule automatically combines with all the other rules about dependencies that I’d written in the analysis.

Who originally created the nodes of the form foo/...? Where are all the places that might consume them? I don’t know! I don’t care! I just let the Datalog engine figure it out.

The curse of brevity

In that blog post, the demonstration seems more blithe than lithe, because we end up showcasing basically two lines of Datalog:

live_out(b, v) :- block_succ(s, b), live_in(s, v).
live_in(b, v) :- (live_out(b, v) ; block_use(b, v)), !block_def(b, v).

Not even that: Our original formulation used only a single rule! We split it into two rules for clarity.

So what’s the big deal? Well, it’s an order-of-magnitude reduction compared to the original code!

def analyze_liveness
  order = post_order
  gen, kill = compute_initial_liveness_sets(order)
  live_in = Hash.new 0
  changed = true
  while changed
    changed = false
    for block in order
      block_live = block.successors.map { |succ| live_in[succ] }.reduce(0, :|)
      block_live |= gen[block]
      block_live &= ~kill[block]
      if live_in[block] != block_live
        changed = true
        live_in[block] = block_live
      end
    end
  end
  live_in
end

The Soufflé examples page has the same problem. They look trivial. Does this look like a substantial program to you?

.decl alias( a:var, b:var )
.output alias
alias(X,X) :- assign(X,_).
alias(X,X) :- assign(_,X).
alias(X,Y) :- assign(X,Y).
alias(X,Y) :- ld(X,A,F), alias(A,B), st(B,F,Y).

.decl pointsTo( a:var, o:obj )
.output pointsTo
pointsTo(X,Y) :- new(X,Y).
pointsTo(X,Y) :- alias(X,Z), pointsTo(Z,Y).

But when I first read that page, I was blown away — only because I already knew how much code it would take in a traditional programming language.

Update 2025-08-23: I see that Hillel Wayne has discussed this in their blog post Instructive and Persuasive Examples (2017). It’s too easy to “brush off” trivial-looking examples of logic programming!

The following are hand-curated posts which you might find interesting.

Date		Title
09 Apr 2017		How to parse contextual keywords in a programming language
24 Jul 2017		Null-tracking, or the difference between union and sum types
23 Dec 2017		Why LINQ syntax differs from SQL, list comprehensions, etc.
20 Apr 2020		Monotonicity is a halfway point between mutability and immutability
14 Jun 2022		Functions are constants too
19 Oct 2022		Build-aware sparse checkouts
17 Jun 2023		Encoding ML-style modules in Rust
02 Oct 2024		Incremental processing with Watchman — don't daemonize that build!
14 Aug 2025	(this post)	Lithe, less analysis with Datalog
23 Aug 2025		Half my work is adding a cache
30 Oct 2025		Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Searching source control graphs

Thu, 24 Jul 2025 00:00:00 +0000

Intended audience	Build system developers and enthusiasts Source control developers and enthusiasts
Origin	Attempts to generalize the library behind `git test` Originally presented at Seattle Build Enthusiasts Meetup (Summer 2025)
Mood	Instructive Enlightened

Table of contents:

Slides
Speaker notes
Related posts
Comments

Slides

This talk was given at the Seattle Build Enthusiasts Summer 2025 meetup (2025-07-22).

In essence: The main idea is implementing parallelizable “generalized binary search” via a structure similar to a comonad, although the talk is not presented that way.

Resources:

Deleted slides with extra information are available at the end.
Direct Google Slides link
Documentation for git test
The alluded-to library is scm-bisect,
- but it’s not production-ready,
- and needs to be updated with e.g. my work on minimization.

Speaker notes

There are some speaker notes in Google Slides. They didn’t correspond to what I actually said.

(1–2) Hi, everyone. My name is Waleed Khan. I’m going to be talking about something that I do every day at work, which is:

(3) Writing buggy code.

(3) I love it. Can’t get enough of it, and the great thing is, since the last time I was here, my ability to churn out buggy code has increased by an order of magnitude, thanks to the power of AI.

(4) So, naturally, I use git bisect a lot. For those of you who aren’t familiar: git bisect is a command that lets you basically binary search for the commit that introduced a certain bug.

(4) Show of hands: how many of you have used git bisect (or equivalent) before?

(4) And have any of you used the ‘automatic’ mode of git bisect run, where you pass a command, and it runs the entire bisection automatically?

(5) I love git bisect because we practice trunk-based development at work, so I have these massive stacks of commits that I’m constantly running tests on. But I had to make it faster.

(6) So I made a tool called git test, which is, confusingly, inspired by Spotify’s tool of the same name. It’s like git bisect, but way cooler. Let’s take a look.

(7) It’s hard to see, but I’m running git test run with cargo fmt to check the codebase formatting, with 8 jobs, and using binary search.

It’s running on 46 commits, and it flags 3 failing commits.
It concludes that the formatting was broken by this buggy commit I inserted into the middle of the stack.
It also checked three different branches, shown here.

(8) So I kept wanting to add more features. And I realized a few things.

Firstly: I’ve invented my own mini-continuous-integration system.
Secondly: Those features look different, but are actually instances of the same problem.
Thirdly: I would really love it if my work’s CI had those features.

(9) How many of you have at least one system in your organization that can automatically bisect build breakages, test failures, or anything else?

(9) Now here’s some slightly different questions.

(9) How many of you have a “merge queue” system to run your continuous integration system on a “pipeline” of commits?

(9) And does anyone have a system that automatically detects and triages flaky tests?

(9) And how about for performance regressions?

(9) Today, I’m going to talk about how to design a library for generalized binary search. My hope is that you can sprinkle some careful abstractions into your CI system and add support for like 400% more new and substantially different workflows with maybe 50% more effort.

(10) The basic idea behind git bisect, and the search problems we’ll see, is the “monotonic assumption”.

(11) Formally, it preserves some ordering relation when you apply a predicate. If element X has a certain property, and element Y comes after X, then Y also has that property.

(11) In practice, this is often true. Thinking of commits, once you commit a bug, it’s usually present in subsequent commits, at least until it’s fixed. So our commit graphs are monotonic with respect to the property of whether a commit has a certain bug.

(11) And the point of this is basically that you can stop your search early. Binary search relies on this in order to guarantee that it finds the first element with a certain property.

(12) I have invented a convenient mnemonic to help you remember this: “Once you begin to fail your predicate, to later pass would be poor etiquette.”

(13–20) (missing: list/discussion of search, minimization, noisy search, and optimization problems that occur in practice)

(21) When I was in elementary school, I learned binary search as “that thing you do in the phone book”.

(22) When I got to college, I learned that binary search is an algorithm operating over a sorted array with worst-case logarithmic time complexity.

(23) After I started working, I learned that binary search is an algorithm operating on monotone predicates over a lattice structure.

(24) Now that I’m older and wiser, I realize that I was overcomplicating things. Binary search is really as simple as 1-2-3!

(25) It’s just the problem of “determining a one-dimensional binary-valued threshold function from a finite class of such functions based on queries taking the form of point samples of the function”, what’s the problem?

(26) Have any of you read Build Systems a la Carte?

(26) It’s a paper that tries to break down various build systems into a bunch of orthogonal axes, then recombine them to create a new build system with various properties.

(26) That’s what I’m going to try to do here. We’ll break down binary search into a bunch of components, and recombine them to solve different problems.

(26) I’m not guaranteeing that this is the perfect design — but hopefully you’ll be able to adapt it to fit your needs.

(27) As an exercise, let’s start by trying to generalize linear search and binary search and abstract out the common details.

(27) So here’s linear search.

(27) And here’s binary search.

(28) (missing: abstracted search algorithm)

(29) Does anyone know how Git selects the next commit to test in this case?

(29) For each commit in the search range, it computes the number of ancestor and non-ancestor commits in that range. It takes the absolute value of the difference, and picks the minimal such commit. That way, it maximizes the information gained in the worst case, just like traditional binary search.

(29) Actually, Git makes the assumption that there’s exactly one buggy commit in the range. If you don’t make that assumption, then it’s technically better to compute the number of ancestors minus descendants. But it’s approximately the same.

(30–31) (missing: generalizing bounds to multiple nodes)

(32) Now, in practice, not every test returns true or false. Sometimes, maybe there’s a compilation error in the codebase, and you can’t run the test at all. In those cases, we want to include an “indeterminate” result.

(33) For linear search, I think it’s pretty clear how to continue searching when you get an indeterminate result. You just skip over that index and continue.

(33) But what do we do for binary search? Does anyone know what Git does?

(33) One reasonable approach is to just remove the commit from the range and pretend it doesn’t exist. The main problem is this heuristic: we’d probably select the parent or child of that commit instead. But there’s a good chance that the build is also broken for that commit. So ideally, we’d pick a commit that’s a little farther away, to maximize the chance that we find a testable commit.

(33) So Git just does the simple and obvious thing here. Which is…

(34) …it generates a pseudorandom number biased according to the square root of the size of the range and uses that offset to select the next commit. Perfectly sane and reasonable, right?

(34) I hope this isn’t contentious, but I’d prefer my search algorithm to be deterministic. We’ll come back to this point with a better solution.

(35) Now, if we wanted to run a bisection system in production, we’d have to scale it up. So, any ideas: how can we run binary search in parallel?

(35) One approach is, at each step, instead of bisection to split the search range into two chunks of approximately equal size, instead we could do n-section to split it into n chunks.

(35) Does anyone see any downsides to this approach?

(35) One problem is that it’s not adaptive. You want to respond to results immediately as you get them; you don’t want to have to wait for all workers to complete. And the amount of parallelism available to you at runtime might fluctuate.

(36) Let’s look at an example:

(36) If we start testing three commits in parallel, and we learn that the first one failed…

(37) …then we want to cancel the other two and start working on the search range to the left of the first one, as soon as any parallelism becomes available.

(38) On the other hand, if the first commit passes, then the new n-section boundaries don’t necessarily overlap with the old ones. If there’s a lot of build overhead to start testing a new commit, then we don’t want to cancel the pending workers and throw away their work, because it’s still useful information, and they’re probably almost done at this point. But we might still want to direct our newly-freed worker to start testing another commit, anyways, so which commit would we pick in that case?

(38) And the other problem is just an engineering problem. The implementation for parallel linear search on a directed acyclic graph is non-trivial, and we’d have to implement it separately from binary search. And we’re actually going to see a bunch of other search procedures in this talk, and it would be painful to have to parallelize each of them individually.

(38) And like I told you, my typical workday looks like this: I head into work, I bang out an incorrect implementation of binary search, and I call it good and then I head home.

(38) It’s a miracle I passed the coding interview.

(38) And it’s not just me. You might have heard that the first published implementations of binary search were wrong for several years, due to unhandled overflow issues. Nobody can write correct code, especially not correct concurrent code, so let’s minimize the surface area here and do this once.

(39) So what’s the trick? Speculation. We’re going to write our code using primitive for non-determinism, and speculate on multiple possible futures at once.

(39) We’re going to distinguish two things:

The Traversal strategy,
and the Search algorithm

(39) and we’re going to put all the speculation logic into the Search in such a way that it’s independent of the Traversal.

(39) Here’s the idea: whenever the Traversal would have us test a certain commit, we’re going to speculate on both the Pass and Fail outcomes, and figure out what we would test next in each future. [slide: speculation tree for binary search]

(39) To parallelize this, all we have to do is take the first n nodes in this speculation tree and start testing them in parallel.

(39) And what’s cool is that, for binary search, this reduces to recursive bisection instead of n-section, which, for powers of two, is equivalent, and for other values, it’s still asymptotically O(log(n)).

(39) And for linear search, this ends up speculating the first n nodes in the range, because the Fail outcomes will terminate the speculation tree immediately, and the Pass outcomes will continue.

(40) The search returns a lazy sequence corresponding to the speculation tree. Internally, it’ll clone the traversal object and speculate on the pass-fail outcomes to produce nodes deeper in the tree.

(41) Each instance of this traversal represents a node in the speculation tree. The speculative search notifies the traversal about a test result, and it updates its internal state, and prunes any unnecessary search nodes and so on, and it returns the updated Bounds for the benefit of the caller.

(41) Then the search queries the “midpoints” of the remaining range. For linear search, those are the next nodes in a breadth-first search in the DAG. For binary search, we compute the ancestor depth to select the midpoints.

(41) Any questions so far? This is one of the main abstractions.

(42) Thinking back to the question of indeterminate commits, we also get a solution for free. Whenever the traversal returns a node that was previously marked as indeterminate, then we can just speculate both pass and fail cases.

(42) For linear search, this again just does the right thing and “skips over” the indeterminate nodes.

(42) And for binary search, we get the ability to select an asymptotically-optimal commit that’s far away from the broken commit, but in a fully deterministic manner. So I think that’s pretty elegant.

(43) Why is it so elegant? Well, let’s look back at the traversal interface. You can duplicate instances of this type; you can extract the current search nodes from the instance; and you can extend the instance into the future by speculating on the search node outcomes…

(43) Eh? I know what you’re thinking. “Wow, that sounds a lot like a comonad!”

(43) Okay, not really, and if any of you were thinking that, then I don’t know if any search algorithm can find relief for your tortured soul.

(44) (missing; re merge queueing)

(45) You can consider normal bisection to be a special case where you can only select sets of adjacent nodes in the graph.

(45) The idea here is to lift the graph such that it operates on combinations of commits, extracted and applied independently, instead of individual commits with all of their ancestors.

(45) Formally speaking, we want to operate over the power set of the search range, which is the set of all subsets of the search range.

(45) The great thing is: the power set naturally forms a directed acyclic graph, where there’s an edge between a parent and a child when the parent is a strict subset of the child. And we just described a scheme to do fancy search on DAGs.

(45) Easy, right?

(46) Okay, so what’s the problem with running the search on this graph?

(46) Normal source control graphs are pretty linear, and have relatively low branching factor. But this graph is maximally dense, with a high branching factor — and it contains an exponential number of nodes.

(46) How do we resolve that?

Avoid materializing entire graph
Make assumptions about search space
Accept a local minimum

(47) (missing: discussion of how to implement conflict analyzer and weighted probabilistic speculation in the same system)

(48–50) (missing: discussion of implementing “delta-debugging” in the same system)

(51) (missing: discussion of minimizing order-sensitive failures)

(52) (missing: discussion of “flake-aware culprit finding”; discussion of weighted-median approach to running multiple tests on the same graph)

(53–55) (missing: going from “generalized binary search” to “noisy generalized binary search” via Bayesian inference)

(56–60) (missing: numerically-incorrect example of updating posterior probabilities, which needs to be fixed)

(61–63) (missing: conclusion and remark on AI-accelerated development increasing the need for clever search systems)

(64–78) (deleted slides with various bullet points)

The following are hand-curated posts which you might find interesting.

Date		Title
24 Jul 2025	(this post)	Searching source control graphs

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Language-learning anecdotes

Sat, 05 Apr 2025 00:00:00 +0000

Intended audience	Those with a passing interest in linguistics.
Occasion	April Cools' 2025 — a little late!
Origin	Personal experience.
Mood	Amused.

These are some anecdotes from my various experiences learning languages (including Urdu, Polish, Latin, and American Sign Language). The order of entries is mostly arbitrary. Hopefully you find some of them entertaining!

Miscellany
Sentence formation
Pronunciation
Grammar
Related posts
Comments

Miscellany

We all scream for ice cream?

Here’s an error screen from Twitter (from many years ago), localized to Polish:

On this error page, the text along with the ice cream cone in the background refer to the rhyme “I Scream, You Scream, We All Scream for Ice Cream”.

The text roughly translates to:

The page isn’t working

I yell. You yell. We all yell… to finally fix this page. […]

Given that the ice cream rhyme relies on wordplay, it seems very likely that this would be unintelligible to Polish speakers.

Regrettably: Even in English, it relies on the reader making a semantic connection with the background image, which is its own accessibility issue.

On hedgehogs

Here’s someone else’s recollection of a story, which I thought was funny:

I was telling my German colleague about how my parents, who were living in Spain, kept having problems with eagles killing their chickens.

Me: yeah, these eagles come down out of the mountains, they steal the chickens right out of the pen!

Frank: an eagle!? AN EAGLE DOES THIS!?

Me: yes, they have eagles over there, mad isn’t it?

Frank: AN EAGLE IS STEALING THE CHICKENS?

Me: errr, yeah, they’re predators you know?

Frank: THEY ARE PREDATORS?

Me: yeah, and they fly down and kill the chickens and fly off with them

Frank: THEY FLY… wait, hang on… Oh…. I see… Err… In German, Igel is… Errr…

(long, long pause)…

AH! IN GERMAN, EIN IGEL IS A HEDGEHOG!

I had assumed that the German word Igel (“hedgehog”) might be a cognate of Polish igła (“needle”), although I didn’t find a direct etymological connection.

Here’s a humorous misspelling in Polish:

A Facebook post that says “Jeżu Ufam Tobie” with an attached portrait of Jesus. The Facebook auto-translation reads “Hedgehog I trust you”.

This misspells Jezu (singular vocative form of Jezus “Jesus”) as jeżu (singular vocative form of jeż, “hedgehog”). Naturally: I’ve adopted “jeżu” as my expletive of choice in Polish.

The Polish word for “porcupine” is jeżozwierz:

jeż + -o- + zwierz.

whose components might translate to “hedgehogobeast”, which I thought was fun.

My family took my wife and I to buy her a lehenga for our wedding celebration. My father advised me that they expected to haggle, as is tradition:

On the buyer’s side: We wanted to avoid giving away information and losing negotiating leverage, so I communicated with my wife in Polish, which only my wife and I spoke.
On the seller’s side: They spoke with my parents in Urdu, which my wife doesn’t speak.
We would switch to English when we all needed to converse.

For a few minutes at a time, I would undergo what I found to be perhaps the most mentally-taxing activity of my life: conducting a many-way negotiation with hidden information across three languages.

To elaborate: I had to translate between Urdu (weak) and Polish (non-native), going through my native English as an intermediary, in close to real-time.

Since my grasp of Urdu is weak, I missed several contextual and cultural clues, and had to explain

not only the facts I understood,
but the inferences I was making
and the confidence I had in those inferences.
But in Polish.

Furthermore: I had to play a careful kind of social deduction game, in which:

I had to keep in mind which information each of the parties had:
- my wife and her judgments;
- myself and my own judgments;
- my and my wife’s joint public statements;
- my family’s public statements;
- and the seller’s public statements;
- what I know my family knows and believes privately, which I have to be careful not to undermine;
- how my family would interpret my and my wife’s joint statements, based on what they know we know.
I had to be sure to convey public information between all relevant parties, while avoiding leaking private information.

For example: Suppose my family were to claim “we’re visiting a few more shops after this”, when in fact we had nothing else on the agenda for the day:

Then I would have to be careful to secretly get my wife to corroborate, in order to avoid undermining our negotiating position,
or I’d have to consider what I was signaling if I chose to contradict my family.

Forgetting my native language

I largely dropped Urdu at home:

My parents would speak in Urdu and I would reply in English.
- This is a common situation in immigrant households. I don’t know if it has a name.
Since they were already fluent in English, over time, they largely switched to English at home.
- Having failed to acquire Urdu well, I also got frustrated easily, or failed to understand some speech entirely,
- which no doubt led to a feedback loop of encouraging English to be spoken at home.

Oops: In university, I took an introductory linguistics course. We all filled out a brief survey, including information about which languages we spoke. By that point, I forgot to fill in ‘Urdu’ 🙃.

Sentence formation

Hacking the Latin word ordering

In university, I took a Latin course:

English uses subject-verb-object (SVO) word order,
while Latin is highly-inflected. The word order is largely free-form, but conventionally subject-object-verb (SOV).

I encountered a great deal of mental friction as I tried to order the sentence components in the conventional SOV manner.

Then I realized — my other native language, Urdu, also has SOV word order. And suddenly it became trivial!

Interestingly: It was as if I mentally rearranged merely the sentence’s concepts in SOV order by passing them through some Urdu circuit in my brain, while relying on my English ability for everything else relating to sentence formation.

To elaborate: I didn’t have enough familiarity or vocabulary to actually translate anything into Urdu as an intermediate step.

So this only leveraged my instinctive syntactic and parsing ability, and nothing semantic.
Just as before: I didn’t get to leverage instinctive noun agreement.
In any event: English vocabulary is much more similar to Latin (via French) than Urdu vocabulary is, so the literal translation of concepts is easier directly from English to Latin.

Concretely: I found SOV a little difficult when thinking in SVO:

In SVO, once you reach the verb, you know you that you’re done parsing the subject’s noun phrase, and can “unwind” it,
whereas in SOV, you might have to build up two complicated noun phrases — and distinguish their boundary — and keep them in mind simultaneously.
I felt that I primarily relied on the syntactic ability to juggle both noun phrases at once.

Making my English worse with Latin

Latin uses participles more often than English, which is described well here. This bled into my “natural” or “default” English, and has led to me using sometimes-unconventional participial phrasing:

For example: “Someone having learned Latin might use participles more often” rather than “Someone who’s learned Latin might use participles more often”.
For example: “With me having learned Latin, I use participles more often” rather than “Given that I’ve learned Latin, I use participles more often”.

Making my English worse with Polish

Learning Polish has made my English more non-standard:

I sometimes build up adjectival or adverbial clauses that are uncomfortable to parse, rather than spreading them throughout a sentence more naturally.
- For example: “I learned Polish, which caused me to more-unusually order adverbs” rather than “I learned Polish, which caused me to order adverbs more unusually”.
I sometimes use verbal nouns when subordinate clauses would be more conventional.
- For example: “My having learned Polish caused me to use verbal nouns unconventionally rather than “I learned Polish, which caused me to use verbal nouns unconventionally”.

Preferring a completely-foreign word order

Similarly to Latin, Polish is a language with conventionally SVO word order, but with enough inflection to support technically free-form word order.

After achieving basic fluency in Polish, I noticed that my preferred word order was most often verb-subject-object (VSO)!

This was unexpected: None of my languages conventionally use VSO, so it’s interesting that I might prefer it, despite having no (grammatical) way to express it in my available languages.

Possibly: It might reflect a desire to express topic-comment word order,

rather than specifically VSO, supposing that I’m most often commenting on actions.
Possibly: My inclination to structure discourse as a top-level comment followed by hierarchical detail might be the same thing.
Likewise for my inclination to
- extract and format mood/uncertainty/provenance signifiers, like “Possibly:”,
- separately from the semantic or factual content of the sentence.
- It’s information that I want to include, but without muddling the syntax.
  - You may disagree that the result is any clearer 🙃.

Note: In Polish, the verb embeds the grammatical number of the subject, which is a fairly common language feature.

Therefore: Saying the verb first also reduces the latency of information transfer, in some sense.
Despite this: I would often still be inclined to include the subject explicitly after the verb.
- Even if it’s a largely-redundant subject pronoun, due to the grammatical number already being embedded in the verb.
For example: I might say “zrobili oni to” rather than just “zrobili to”.
- “zrobili”: “to do” third-person plural past-tense perfective
- “oni”: “they” third-person plural
- “to”: “it”/”this” third-person singular
For a native speaker, I expect that it would be interpreted as emphasizing the subject much more strongly than I intended it?

Failing to communicate privately in American sign language

My wife and I learned a few signs of American sign language (ASL).

Originally: I thought that ASL could be useful when traveling:

Particularly: In loud or distant contexts, such as restaurants, parties, public transit, sightseeing, and so forth.
For example: It can be convenient to communicate from across a museum hall my intention to go to the bathroom, so that my partner isn’t confused about where I’m going, and knows that I’ll return shortly.
For example: It’s great for excitedly and efficiently summoning the other partner to come see a cute local cat or dog which might soon leave sight.

We’re often able to speak in Polish to communicate privately, but that’s not an option if we happen to be around other Polish speakers, or if we simply can’t hear each other. So I thought it might be convenient to sometimes use ASL for private communication.

But it turns out: Communicating a desire to leave the current venue by gesturing at ourselves and then pointing at the exit with both index fingers is not a terribly polite way to coordinate our departure from a friend’s event.

Pronunciation

Phonological production in Polish, but not distinction

Polish has two phonologically-distinct ‘ch’ sounds, as opposed to English’s one:

The two sounds are usually written ‘cz’ and ‘ć’
- corresponding to IPA /t͡ʂ/ and /t͡ɕ/
- although with some additional orthographic rules, such as ‘ć’ sometimes being written ‘ci’.
Polish also has two ‘sh’ and two ‘zh’ sounds with the same phonological relationships.

I can produce these sounds reasonably well, but I nonetheless can’t reliably distinguish them:

When I hear a new word that uses one of the sounds, I usually ask the speaker “is it (version with cz) or (version with ć)?”.
- It doesn’t help that the word for “or” in these questions is ‘czy’, involving another instance of the ‘ch’ sound.
Then they repeat back one of the forms to me, after which I realize I can’t distinguish it anyways.
Then I give up and either ask them to spell it, or I look up the word later.

This is unsurprising from a linguistic perspective, but amusing to me nonetheless.

Subvocalizing written Chinese

I was briefly learning written Chinese, but not spoken Chinese.

(I have fun learning and writing the characters, but I don’t have anyone to speak Chinese with.)
Consequently: I didn’t have names or vocalizations for any of the characters, beyond some of the radicals.

So what did I do when I happened to subvocalize characters while practicing? I would just pronounce the Polish word for the same concept!

For example: “水” would become “woda” rather than “water” or “shuǐ”.

Apparently: I just compartmentalize my vocabulary into “native” and “foreign”, without much nuance beyond that!

Adjusting my English accent

At work, we once had several visitors from my team’s other offices. In one instance, at lunch, we had a majority of people with British or Australian accents rather than American accents (perhaps 3–4 vs 2).

Somewhat embarassingly: Once the conversation seemed primarily British-/Australian- rather than American-accented, I found myself adopting their accents without voluntarily deciding to do so 😬:

I had said e.g. “queue” instead of “line”.
I switched to using rising inflection for my questions rather than falling inflection,
Nobody else at the table commented, so I don’t know if they noticed, but perhaps it came off as pretentious 🙃.
I don’t recall this ever happening again for my English.
- At least: Not to such an alarming degree, anyways. I do naturally adopt various other speech patterns, code-switch, etc.
- My team lead at the time (whom I saw most days, as was normal in the Before Times) had a British accent:
  - So I had probably ingested it passively over time,
  - but I’d never voluntarily or involuntarily mimicked it.

Making others’ Polish accents worse

In English, out of enthusiasm, I’ll often lengthen certain vowel sounds, such as “hi-iii” or “tha-ank you-uu”. Without consciously considering it, I carried over the same behavior to Polish, such as “cze-eść” (cześć, “hello”) or “dzięku-uję-ę” (dziękuję, “thank you”).

My wife and brother-in-law have informed me that this is simply not done in Polish, and, in fact, I have started to poison their own speech patterns 😈.

Failing to improve my Polish accent

I learned Polish at a basic level to communicate with my wife’s family. When speaking to them in Polish, particularly her parents, I don’t tend to pick up their accent or pronunciation.

But, when abroad, dining with my wife’s cousins (and their respective dinner partners), I noticed involuntary effort on my part to match their pronunciations. I don’t know what the difference is.

To conjecture: It’s hard to decide whether it’s primarily due to low engagement (and compensating to make up for it) or high engagement (encouraging me to absorb more). Maybe a linguist would know. Some ideas:

I think my wife’s parents speak more quickly than average. It tends to be harder for me to follow/understand them.
- Maybe: I can’t “hear” the pronunciation as well when the speech is quicker.
- Maybe: I need to expend more mental effort on semantic comprehension, and have less available mental throughput to spend on comprehending and analyzing our styles, and matching mine to theirs.
Maybe: I talk about more engaging or interesting topics (to me) with people my own age, somehow motivating me to analyze the pronunciation more.
Maybe: I’m naturally more engaged with unfamiliar pronunciations and accents, and perhaps don’t “hear” familiar pronunciation/accents anymore.
Maybe: Some specific accents are easier for me to analyze and match.
Maybe: The accents of Polish speakers in America have integrated the ambient phonology somewhat, thus causing the accents to stand out less to me, to the point where I don’t feel the inclination to match them.

Grammar

Urdu gender confusion

I grew up in Michigan, with my immediate family having settled there. I was raised with primarily English and secondarily Urdu as native languages.

Additional background: I always spoke Urdu haltingly:

Both of my parents spoke English fluently, but would usually speak Urdu at home.
My vocabulary was very limited,
but I largely understood the syntax in that intuitive first-language way.
- Presumably due to having acquired it during the critical period.
- My brain will parse utterances I overhear into syntax trees “in the background”:
  - without deliberately choosing to expend mental effort,
  - and despite the fact that I don’t understand almost all of the words.
- This is unlike Polish, which I learned as a second language to a basic conversational level.
- Note: While I had a solid grasp on the syntax, I had a poor grasp of the grammar, which involves semantic agreement; see later.
I was never literate in Urdu.

My mother would take my sister and I to visit our extended family in Pakistan over summer break:

Everyone lived in one large home and spoke Urdu.
During the day:
- The women would stay at home to do the housework and childcare, or leave for errands.
- The children would stay at home. The other children were girls (my sister and cousin).
- The men would commute to their respective workplaces in the morning and return in the evening.
  - One of my uncles would actually stay in another city during the workweek, and only come home for the weekends! That seems uncommon in the States.

Consequently: The majority of my language-acquisition time was spent among exclusively women.

In Urdu, verbs are conjugated according the subject’s grammatical gender (but see below). Despite my having masculine pronouns, I primarily observed speakers with feminine pronouns, and hence I learned all of the feminine forms instead.

That is: I only knew how to refer to myself as a woman!

To elaborate: Here’s some more details on the grammar:

For example: The first-person singular pronoun is always “mai”. But to say “I do”, one would say either “mai karta hun” (masculine) or “mai karti hun” (feminine). I only learned the latter form.
Furthermore: Urdu has split ergativity, so, under specific conditions, the verb agrees with the object instead of the subject.
- This would have been completely inscrutable to me as a child.

Truly: The poverty of the stimulus lays in shambles.

Polish gender confusion

Amusingly: Like Urdu, Polish also conjugates some verbs according to the speaker’s gender. Also like Urdu, I mis-trained the gender of some verb forms based on my available practice partners!

I learned/trained primarily with my now-wife, as well as her Polish friends who lived in the same building as us.
Consequently: I overtrained the feminine second-person verb forms, and had to untrain them to speak correctly when encountering a non-feminine audience.
- Unlike Urdu, I knew the correct grammar. I had just built incorrect habits and heuristics.

Urdu grammar confusion

Despite having acquired some facets of Urdu as a first language, I failed to acquire several critical features:

For example: I don’t have any instinctual noun classification/agreement beyond what exists in English,
which is unfortunate, because it might have been nice to automatically have instinctive noun agreement in other languages, such as Spanish.

For whatever reason: My younger sister learned Urdu more fluently than I did.

The following are hand-curated posts which you might find interesting.

Date		Title
24 Aug 2016		Steno Journal: Weeks 1-12
06 Sep 2016		Stenography adventures with Plover and the Ergodox EZ, part 1
06 Sep 2016		Stenography adventures with Plover and the Ergodox EZ, part 2
13 Oct 2016		Analyzing all of my Messenger conversations to create conversational macros
21 May 2018		My steno system
02 Jan 2023		On bullet points
11 Jan 2023		Interactive blogs
05 Apr 2025	(this post)	Language-learning anecdotes
03 Apr 2026		🙏: please or thank you?

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Incremental processing with Watchman — don't daemonize that build!

Wed, 02 Oct 2024 00:00:00 +0000

Intended audience	Build system developers and enthusiasts Developers of tooling that doesn't fit into the build system
Origin	Lessons synthesized from 8-year career in incremental computation and build systems Originally presented at Seattle Build Enthusiasts Meetup (Fall 2024)
Mood	Instructive Passionate that people should avoid my mistakes

Table of contents:

Slides
Speaker notes
Related posts
Comments

Slides

This talk was given at the Seattle Build Enthusiasts Fall 2024 meetup (2024-10-02).

Deleted slides with extra information are available at the end.
Direct Google Slides link: https://docs.google.com/presentation/d/1geE7rGTCpIk5UTjH-sw9aL_Zszi1ncCRnFi55t0CDlk/

Speaker notes

I’ve transcribed the speaker notes below, which correspond to the intended transcript.

(1) Hi everyone, thanks for attending. My name is Waleed Khan and today I’ll talking about building incremental systems on top of the Watchman file-watching service.

(2) If you want to follow along with the slides at your own pace, you should take this opportunity to open them up now. You can visit my website waleedkhan.name and you can navigate to the blog post with these slides.

(2) A little about me: I used to work at Meta on the Hack programming language, and Twitter on their Scala build tooling, as well as source control. Most recently, I’ve been working at Hudson River Trading on their C++ and Python build system.

(2) My career has been spent shortening the developer feedback loop, largely by implementing custom incremental build systems. Watchman is an open-source file-watching tool that can be used as a high-level primitive for those kinds of incremental systems. In this talk, I’d like to share some design ideas for implementing similar incremental build systems.

(3) There’s two parts to this talk.

(3) In the first part, I’ll talk about the justification for Watchman, the problems it solves, and the abstractions it offers.

(3) In the second part, I’ll just be listing several helpful tips and pieces of advice for implementing systems on top of Watchman.

(4) Let’s give an example of what kind of problems might be solved by Watchman.

(5) I mentioned I’ve spent a lot of time working on custom incremental build tooling. When would you need that? I’ll give a few examples that I’ve personally used Watchman for in order to implement custom incremental build systems.

(5) One example is file syncing. A lot of companies have been embracing “remote development” workflows, by which I mean your code is sent from your local machine to be compiled and run on a remote machine somewhere else. Many companies have invented solutions to sync your local source code to the remote machine as part of the development process. File syncing may be outside the scope of your build system, or it may just not be integrated yet.

(5) Another example is dynamic dependencies in the build system. Many companies have Bazel, but those kind of build systems may not be able to express certain kinds of dynamic dependencies. For Bazel, there’s a tool called “Gazelle” which is used to programmatically generate BUILD files, which are then read by Bazel in the next build. There may be some fans of the buck2 build system in the audience who scoff at this, given its native support for dynamic dependencies. [We can’t all use buck2, unfortunately.]

(5) Less common, but still important, is building latency-sensitive IDE services. In those cases, the build system may not be able to generate the appropriate build artifacts, possibly because of dynamic dependencies, or the overhead may be too much to serve queries in real-time as the code changes. In those cases, more specialized infrastructure is necessary.

(6) For demonstration, I cloned the “nixpkgs” repository from GitHub. It’s not a huge repository by industry standards, with 43k files in the working copy, but it’s enough that walking the entire repository is slow on my machine.

(7) For example, on my machine, just visiting all the files in the repository without reading them takes around 750ms. And that’s without any additional processing. If you wanted to actually perform a build, you could expect significantly more overhead.

(8) If we were doing remote development, we might simply use a tool like rsync to efficiently sync all the local files to the remote machine.

(9) If we just naively run rsync on my machine, from one directory to another, it takes around 1.4 seconds for a no-op sync. That is, if I haven’t changed any files, it takes 1.4 seconds to do nothing. That’s because rsync has to walk and hash all files on the local and remote sides to avoid transferring data that’s already present on the remote.

(10) So what can we do? I’ll give a few ideas.

(10) One is to maintain a persistent data store to try to avoid reading files that probably haven’t changed, similarly to how Git maintains its index. This is helpful but still scales linearly with the size of the repository, so it’s not necessarily sufficient by itself.

(10) Another is to launch a background process (a “daemon”) that waits for the operating system to notify it about changed files, and then syncs the files preemptively. This is a workable, but I’ll argue hugely complex, solution.

(10) Now I have to explain the subtitle for this talk, “don’t daemonize that build!”.

(11) By “daemon”, I basically mean a background process that runs independently of explicit user interaction. Oftentimes, I see daemonization proposed as a natural next step when a build task is starting to take too long to run end-to-end.

(12) There are a few common reasons to want to daemonize.

(12) One is to keep previously-computed data in memory without having to recompute it or reload it. We’ll discuss some ways to address that without a daemon in this talk.

(12) Another one is just the fact that operating system APIs require that you subscribe to filesystem notifications in a long-running process in order to actually receive the events as they happen. We’ll discuss how Watchman fits in here.

(12) One last thing is specifically to reduce startup latency due to the choice of programming language or runtime. This talk unfortunately doesn’t address that issue.

(13) So why not daemonize? I’m not going to read off every point on this slide, but, basically, it’s a ton of incidental complexity, by which I mean complexity you have to deal with that’s not really helping you accomplish your engineering goals.

(13) Daemonization is an attractive solution to improve performance at first glance, but it incurs a lot of unexpected and insidious costs down the road.

(14) So my recommendation is: avoid daemonizing your build tasks for as long as possible.

(15) This talk is about using Watchman, a file-watching service, as a way to abstract out a lot of the incidental complexity and help you focus on tackling your actual engineering problems.

(15) You can use Watchman as a high-level primitive in custom build systems. Watchman is a great tool in my opinion because it can completely change the design of your system for the better. And, in fact, even if you do one day decide to daemonize your system, you may benefit a lot from continuing to use Watchman and designing your system around the abstractions that it offers.

(15) I’ll also mention that other systems offer similar interfaces. For example, the build system buck2 supports a similar interface via its “incremental actions”, and the Language Server Protocol supports subscribing to file-watching notifications.

(16) So let’s get started with Watchman.

(17) The first thing we have to do is ask Watchman to start watching a directory with the watch-project command. Easy enough.

(18) Next, let’s issue a basic query.

(18) Notice that we’re calling Watchman via the command-line interface with a JSON payload. This is already a huge improvement from an engineering perspective, just because it’s way easier to experiment with this rather than, for example, the inotify syscall interface for Linux in a long-running C program.

(18) Here’s, I’m piping in a JSON payload to standard input using bash. The subcommand is query, the watched directory is the current directory, which is nixpkgs in this case, and the last parameter is the query options.

(18) By default, if we don’t pass any query options, Watchman returns data about all of the files in the watched directory. In this case, I piped the result to jq to just get the first file for demonstration purposes. We’ll see how to do incremental processing shortly.

(18) In the result, you can see the filename — in this case, it’s happens to be the .git directory — and a few other metadata fields that are returned with the default query options. You can ask Watchman for a different set of metadata if you want.

(19) Now let’s look at the metadata returned by Watchman for the query itself, rather than the metadata for individual files.

(19) There’s two main things to look at. The first one is clock, which is the key to working with Watchman. Basically, this value represents the point in time immediately after the query returned. I can use this value in subsequent queries to get a list of changed files since that previous clock ID or point in time. It’s very similar to a database cursor in that respect.

(19) The second one is is_fresh_instance, which basically means that I, the caller, may have missed some filesystem notifications since the last time I queried Watchman. We’ll discuss the implications of this in more detail later.

(20) To use the clock ID, you can add since to your query with your previous clock ID, and Watchman will only return files that have changed since that clock ID.

(20/21) In this example, I made an initial query and got an initial clock ID, and then I immediately use it in the next query via the since field, and Watchman returns an empty files array, meaning that no files have changed since that clock ID. Then I create a new file called foo and query since the same clock ID as before, and this time Watchman returns the file foo.

(22) Watchman will also tell you about files that have been deleted since the clock ID. You can see here that exists is now set to false, meaning Watchman thinks that the file has been deleted on disk since the last clock ID. We often need this for the correctness of build tasks in order to invalidate data corresponding to deleted files.

(23) And just a few more features. Here, we’re explicitly setting metadata fields we’re interested in, including the SHA1 content hash for each file. If you want, Watchman can compute and store the content hashes for you in memory, which can be convenient for caching as discussed later.

(23) And we can provide arbitrarily complex expressions to limit what kinds of entries Watchman returns. In this case, we’re limiting the returned entries to files of type f, which is to say that they’re real files, and not directories or symlinks or something else, mainly so that I can show an example of Watchman returning the content hash. And we’re excluding files under the .git directory, so that Watchman returns a real source file instead of some random Git metadata file.

(23) In the result, we can see that Watchman returned some Main.java file along with its metadata and hash as the first file.

(24) So let’s show a real example of working with Watchman.

(25) This is a bash script to implement incremental rsync. Instead of unconditionally syncing all files, we’ll sync only the files that have changed since the last sync.

(25) First, we start watching the project. We load the previous clock ID from persistent state on disk, or a default clock ID if it’s not available. We query Watchman, and based on is_fresh_instance, we do either a full sync or an incremental sync.

(25) We have to do a full sync if it’s the first sync, or if we missed any filesystem updates, to bring the remote side back to a known state. That includes deleting any extra files on the remote side with rsync’s –delete-after option.

(25) In the incremental sync case, we limit rsync to syncing only the changed set of files. Note that this set may include deleted files. We pass –delete-missing-args to rsync to indicate that a missing file is not an error, but that we actually want rsync to delete that file from the remote side.

(25) Then we run rsync, and we make sure to only save the new Watchman clock if it succeeds. You can think of this as similar to “committing” a database transaction. If the rsync failed for some reason, then the next attempt will try to sync all of the old changed files again, plus any newly-changed files.

(26) If we try it out, you can see that it’s around 4.5x times faster than the naive rsync solution, with not a lot of development effort. And I want to emphasize that we could definitely improve on the efficiency of the script.

(27) Hopefully you’re starting to see the utility of Watchman as a primitive for your build tasks by now.

(28) There’s much more detail about Watchman advantages and disadvantages in the online slides, but I’ll just include this snippet from a blog post by an engineer at Stripe. [read quote aloud]

(29) The real advantage of Watchman is that encourages you to design and implement correct systems.

(29) If you think about it, your filesystem is a kind of distributed system. You can send messages to files in your filesystem and ask them about their current state, but it’s always possible that someone else will come and modify a file when you’re not looking. So, fundamentally, it’s much harder to work with a filesystem than just a hash map in memory, for example.

(29) The Watchman design lifts the level of abstraction from “synchronizing against a shared mutable data store” to “processing an ordered, replayable stream of events”. And that’s the big advantage of using Watchman. It’s just a nice bonus that it has a bunch of other practically-useful features for building incremental systems.

(30) I’ll talk a little bit more about a couple of features of the Watchman design.

(31) Practically-speaking Watchman supports three main workflows:

(31) One is where you run a program which queries and processes changes on-demand, which is the model we’ve been talking about so far, and that can take you a long way.

(31) But with the trigger subcommand, you can invert the control flow and ask Watchman to call you when something changes. This is very useful if you want to process changes immediately, instead of when the user asks you to. Watchman will ensure that there’s only one instance of your command running at once, and it can also do things like batch changes for efficiency.

(31) Or you can subscribe to changes in a long-running process, similarly to using the operating system APIs, but at a higher level of abstraction. So you can still use Watchman with a daemonized build.

(32) Underneath the hood, you can consider Watchman as maintaining a linked hashmap in memory, from path to file metadata. When Watchman receives a filesystem notification for a path, it looks up that entry, updates the metadata, and bumps it to the head of the list in constant time. When you ask Watchman for recent changes, it iterates the list and returns entries until it reaches one older than the clock ID you provided, then terminates the traversal.

(33) Next, I’ll shift gears to giving practical advice to using Watchman. But before that, I’d like to take any questions that you might have.

(34) First, I’ll discuss some rules to follow in order to build correct systems on top of Watchman.

(35) I recommend you read the linked Stack Overflow answer on what is_fresh_instance means. Basically, you need to assume that any persistent state may now be invalid. You have to either clear the persistent state or validate it before using it again.

(36) Be mindful when using queries that might change. One example is if you construct a query using .gitignore files in the repository, you may have a cyclic dependency between .gitignore files and the query that watches those .gitignore files for changes. If you’re not careful, you might end up in a situation where you can’t detect certain changes to ignored files.

(37) Like we discussed earlier, the local filesystem is basically a racy distributed system. For example, if Watchman reports a changed file, you may try to read it from disk, and discover that it’s been deleted since the last Watchman query. You should plan to handle that case up-front.

(38) Now some tips and common patterns for using Watchman.

(39) One quick algorithmic tip. I recommend that you try to unify your full and incremental processing paths as much as possible. The way to do that is to treat Watchman results not as a set of files to traverse in the filesystem, but instead a set of files to limit your traversal of the filesystem. Here’s an example code snippet: we put all the changed paths into the set paths_to_visit, plus their transitive ancestor directories. If we got is_fresh_instance from Watchman, then we set paths_to_visit equals None and don’t limit traversal at all.

(40) Then, in the filesystem traversal code, we just check should_visit before processing each path, to ensure that we skip processing most files in the incremental case. It really helps improve the reliability of your incremental system when you have fewer distinct code paths to exercise.

(41) If you want to store persistent state, then you probably want a mapping from path to metadata, with the ability to update — and invalidate —individual key-value pairs efficiently.

(41) An easy way is to just store a big JSON blob on disk, read it on startup, make any changes in-memory, and write it on shutdown. This can still be much faster than processing the entire repository contents every time.

(41) Cleverer alternatives include SQLite, which an on-disk relational database, and RocksDB, which an on-disk key-value store.

(41) I’ve used all three of these approaches.

(42) You may need to compute data that’s not indexed directly by path. For example, in an IDE, you may take a file path as a key and compute the list of symbols defined in that file as the value. To support go-to-definition, you would want a mapping in the other direction, from symbol name back to file path.

(42) Maintaining the reverse mapping is just a general build system problem at this point. One technique I want to point out is that it’s oftentimes feasible to not fully update the reverse mapping and continue to store stale edges. For example, if I store a reverse mapping that says that class Bar is defined in path foo.py, and then I delete foo.py, I might not update the reverse mapping at all. Instead, when I query the path associated with class Bar, my code could actually go re-parse file foo.py and confirm that class Bar is still there before returning it as a result.

(43) For performance, if you want to keep your persistent state alive across is_fresh_instances, you can use an extra key to confirm that data is still valid. One idea is to store the content hash associated with a file, so that you know that the data is still valid without needing to reprocess the file.

(43) If you didn’t clear the entire persistent state when getting is_fresh_instance, then keep in mind that the set of paths in your persistent state might include extra stale paths if you don’t explicitly clean them up.

(44) If your caching scheme is machine-independent, and your build artifact is expensive enough to compute, then it might make sense to rely on a recent precomputed artifact or service as a base, and compute the changes since then. Supposing that I have a recent build artifact at tag warm, I can download it, and then ask source control for the files changed between the warm commit and HEAD, and rely on Watchman from that point onwards. This is what I did to support Hack IDE services at Meta.

(44) To do this correctly, make sure you have a strategy for handling files deleted since the warm tag. For path-addressable data stores, you can employ a technique like tombstones. For content-addressable data stores, you might effectively handle deletions for free.

(45) That’s all of the practical items. Thanks everyone for watching! Hopefully you learned something about incremental computation or will consider using Watchman to avoid daemonizing your next build. If you’re following along with the slides, you can go to the next slide to see some of the content I cut.

(45) If you enjoyed this talk, I’m on the job market currently and would love to continue working in the developer infrastructure and tooling space. You can reach out to me in person or at my email, which is me@waleedkhan.name.

(45) Now I’ll open up the floor to questions.

[no notes for deleted slides]

The following are hand-curated posts which you might find interesting.

Date		Title
20 Apr 2020		Monotonicity is a halfway point between mutability and immutability
19 Oct 2022		Build-aware sparse checkouts
02 Oct 2024	(this post)	Incremental processing with Watchman — don't daemonize that build!
14 Aug 2025		Lithe, less analysis with Datalog
23 Aug 2025		Half my work is adding a cache
30 Oct 2025		Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Patch terminology

Thu, 11 Jan 2024 00:00:00 +0000

Intended audience	Developers of version control systems, specifically jj. Those interested in the version control pedagogy.
Origin	Reprint of research originally published on Google Docs. Surveyed five participants from various "big tech" companies (>$1B valuation).
Mood	Investigative.

Table of contents:

Methodology
Results
Remarks
Conclusions
Related posts
Comments

Methodology

Q: I am doing research for a source control project, can you answer what the following nouns mean to you in the context of source control, if anything? (Ordered alphabetically)

a “change”
a “commit”
a “patch”
a “revision”

Results

P1 (Google, uses Piper + CLs):

Change: a difference in code. What isn’t committed yet.
Commit: a cl. Code that is ready to push and has a description along with it
Patch: a commit number that that someone made that may or may not be pushed yet […] A change that’s not yours
Revision: a change to a commit?

P2 (big tech, uses GitLab + MRs):

Change: added/removed/updated files
Commit: a group of related changes with a description
Patch: a textual representation of changes between two versions
Revision: a version of the repository, like the state of all files after a set of commits

P3 (Google, uses Fig + CLs):

Change: A change to me is any difference in code. Uncommitted to pushed. I’ve heard people say I’ve pushed the change.
Commit: A commit is a saved code diff with a description.
Patch: A patch is a diff between any two commits how to turn commit a into into b.
Revision: Revisions idk. I think at work they are snapshots of a code base so all changes at a point in time.

P4 (Microsoft, uses GitHub + PRs):

Change: the entire change I want to check into the codebase this can be multiple commits but it’s what I’m putting up for review
Commit: a portion of my change
Patch: a group of commits or a change I want to use on another repo/branch
Revision: An id for a group of commits or a single commit

P5 (big tech, uses GitHub + PRs):

Change: your update to source files in a repository
Commit: description of change
Patch: I don’t really use this but I would think a quick fix (image, imports, other small changes etc)
Revision: some number or set of numbers corresponding to change

Remarks

Take-aways:

Change: People largely don’t think of a “change” as an physical object, rather just a diff or abstract object.
- It can potentially range from uncommitted to committed to pushed (P1–P5).
- Unlike others, P4 thinks of it as a larger unit than a commit (more like a “review”), probably due to the GitHub PR workflow.
Commit: Universally, commits are considered to have messages. However, the interpretation of a commit as a snapshot vs diff appears to be implicit (compare P2’s “commit” vs “revision”).
Patch: Split between interpretations:
- Either it represents a diff between two versions of the code (P2, P3).
- Or it’s a higher-level interpretation of a patch as a transmissible change. Particularly for getting a change from someone else (P1), but can also refer to a change that you want to use on a different branch (P4).
- P5 merely considers a “patch” to be a “small fix”, which is also a generally accepted meaning, although a little imprecise in terms of source control (refers to the intention of the patch, rather than the mechanics of the patch itself).
Revision: This is really interesting. The underlying mental models are very different, but the semantic implications end up aligning, more so than for the term “commit”!
- P1: Not a specific source control term, just “the effect of revising”.
- P2, P3: Effect of “applying all commits”. This implies that they consider “commits” as diffs and “revisions” as snapshots.
- P4, P5, Some notions that it’s specifically the identifier of a change/commit. It’s something that you can reference or send to others.
- Surprisingly to me, P2–P5 actually all essentially agree that “revision” means a snapshot of the codebase. The mental models are quite different (“accumulation of diffs” vs “stable identifier”) but they refer to the same ultimate result: a specific state of the codebase (…or a way to refer to it — what’s in a name?). This is essentially the opposite of “commit”, where everyone thinks that they agree on what they are, but they’re actually split — roughly evenly? — into snapshot vs diff mental models.

Conclusions

Conclusions for jj:

We already knew that “change” is a difficult term, syntactically speaking. It’s also now apparent that it’s semantically unclear. Only P4 thought of it as a “reviewable unit”, which would probably most closely match the jj interpretation. We should switch away from this term.
People are largely settled on what “commits” are in the ways that we thought.
- There are two main mental models, where participants appear to implicitly consider them to be either snapshots or diffs, as we know.
- They have to have messages according to participants (unlike in jj, where a commit/change may not yet have a message).
  - It’s possible this is an artifact of the Git mental model, rather than fundamental. We don’t see a lot of confusion when we tell people “your commits can have empty messages”.
  - I think the real implication is that the set of changes is packaged/finalized into one unit, as opposed to “changes”, which might be in flux or not properly packaged into one unit for publishing/sharing.
Half of respondents think that “patch” primarily refers to a diff, while half think that it refers to a transmissible change.
- In my opinion, the “transmissible change” interpretation aligns most closely with jj changes at present. In particular, you put those up for review and people can download them.
- I also think the “diff” interpretation aligns with jj interpretation (as you can rebase patches around, and the semantic content of the patch doesn’t change); however, there is a great deal of discussion on Discord suggesting that people think of “patches” as immutable, and this doesn’t match the jj semantics where you can rebase them around (IIUC).
- Overall, I think “patch” is still the best term we have as a replacement for jj “changes” (unless somebody can propose a better one), and it’s clear that we should move away from “change” as a term.
“Revision” is much more semantically clear than I thought it was. This means that we can adopt/coopt the existing term and ascribe the specific “snapshot” meaning that we do today.
- We already do use “revision” in many places, most notably “revsets”. For consistency, we likely want to standardize “revision” instead of “commit” as a term.

The following are hand-curated posts which you might find interesting.

Date		Title
19 Jun 2021		git undo: We can do better
12 Oct 2021		Lightning-fast rebases with git-move
19 Oct 2022		Build-aware sparse checkouts
16 Nov 2022		Bringing revsets to Git
05 Jan 2023		Where are my Git UI features from the future?
11 Jan 2024	(this post)	Patch terminology

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

Testing terminal user interface apps

Sat, 30 Dec 2023 00:00:00 +0000

Intended audience	Software engineers developing interactive terminal applications.
Origin	Work notes published internally. My experience developing several TUI applications.
Mood	Instructive. I have a lot of work notes that I'd like to publish more widely henceforth.

Introduction

Terminal user interface (TUI) applications are an efficient way of surfacing data in a visual and interactive manner while reusing significant portions of an existing program which normally prints to stdout/stderr. However, bugs in TUI applications can quickly cause users to lose confidence in the visualization tools. This article details methods for testing TUI applications.

Table of contents:

Introduction
Primer on TTYs and PTYs
- Escape codes
- Terminal emulation
Testing patterns
Related posts
Comments

Primer on TTYs and PTYs

A “TTY” is a “teletype” (or “teletypewriter”) and a “PTY” is a “pseudo-teletype” (or nowadays “pseudo-terminal”). In the earliest days of Unix, users interfaced with the computer primarily via a TTY, but these days, the TTY is usually more of an abstraction over the terminal’s visual interface. See The TTY demystified for more details.

The TTY abstraction is actually part of the operating system. The TTY devices available to programs can be found as the special files of the form /dev/tty*. You can determine if a file descriptor refers to a TTY using the isatty syscall. This function call is usually available in non-C languages, such as via Python’s os.isatty.

TTYs can be queried for information such as its dimensions via ioctl. It’s helpful to have a library to wrap querying and rendering to the TTY, or which offers a higher-level abstraction for the terminal UI.

When running automated tests, a TTY is probably not available (or if it is, it would be the TTY for the person who is running pytest, etc., which would not be good to use because other programs are already rendering to it). Instead, you could construct a PTY (with the opentty(3) syscall) or with a higher-level library.

Escape codes

Communication with a TTY is strictly by sending and receiving bytes, so how does the program actually draw a terminal interface on the TTY? It does so by sending special escape codes, such as ANSI escape codes. These are special sequences of bytes which can indicate things like “move the cursor to (X, Y)” or “enable bold and red coloring”.

The VT100 was a popular physical teletypewriter device which implemented the ANSI escape codes. You may sometimes see the ANSI escape codes referred to as VT100 escape codes.

Terminal emulation

Since we don’t use VT100s anymore, we instead use terminal emulators: programs which implement the reading/writing of the TTY device and rendering on screen. The basic set of ANSI escape codes is handled by every serious terminal emulator program (Terminal.app, iTerm2, PuTTY, Alacritty, etc.). You might also say that these terminal emulators are “VT100”-compatible.

Some terminal emulators support features outside those in the ANSI escape code set. For example, iTerm2 can render inline pictures (as well as many other terminal emulators).

When we construct a PTY, we only get a bidirectional stream of bytes, not an actual picture of the screen. In automated testing, we therefore have to interpret all the escape codes and render the screen contents ourselves. To do so, we can use a terminal emulation library, such as pyte for Python or portable-pty for Rust. Typically, we would write a small “expect”-style framework on top of this to test the target application (see “Expect”-style testing).

Testing patterns

Integration vs end-to-end

After having spent some time testing TUI apps, I find there are two general categories of testing:

Integration testing:
- Allows us to hook into the application event loop to inject synthetic events. Examples:
  - Execute a function and synchronously wait for it to complete.
  - Take a “screenshot” of the application at the current time.
  - Specify non-deterministic event data (such as an event that renders timing data).
- It is highly desirable to structure such TUI apps as event-driven for testing purposes!
- Input can be at the user level (e.g. simulate “down arrow keypress”) or at the application level (e.g. trigger “open menu X”).
  - With user-level input, the interface itself is tested more completely, but it can make it difficult to write tests which involve complicated maneuvers (e.g. “scroll down to the third item representing menu X and then press enter”), and such tests will be more brittle when the interface changes.
- Can choose to omit or sanitize non-deterministic data from the interface if necessary (e.g. timing data).
- Can mock out certain facilities as desired (e.g. do not trigger an actual build from the interface).
- Allows us to redirect rendering to a “virtual canvas” instead of rendering directly to the TTY/PTY. This is not strictly necessary (see end-to-end testing approaches with a PTY) but can simplify the testing interface (no need to use a library/write complicated TTY logic; taking “screenshots” will likely be easier to implement).
End-to-end testing:
- Simulate a program under test using only normal user input methods, such as keystrokes.
- Typically uses a PTY to simulate the interface. More discussion on TTY/PTY later.
  - Example: see script(1) for a tool which simulates the terminal session and can play it back later.
- May be difficult to prevent side-effectful operations from occurring (e.g. triggering a build from the interface).
- May need to sanitize the screen contents to remove non-deterministic data (e.g. timing data).
- Not possible to wait synchronously on internal operations, which requires some sort of external indication when an internal operation is complete, and tends to be flakier.

Generally speaking, integration testing will be more direct and less flaky, but end-to-end testing can be a good solution for very broad coverage of a program and to test programs which aren’t currently set up to support integration testing.

Snapshot testing

NB: This style of testing is also called “golden” testing and “expect” testing, along with several other names that don’t come to mind right now.

This is a category of testing where the output from a function or program is gathered and compared to a known-good “snapshot” of that output. A “snapshot” can refer to simple textual data, structured data, or even actual snapshots of the UI of a program. Snapshot testing libraries typically come with a tool to quickly compare and update snapshot output.

Example: the cargo-insta library for Rust implements snapshot testing.
Example: the VCR.py library implements snapshot testing for network requests specifically.
Example: the Jest library for Javascript implements snapshot testing.

There are several advantages to snapshot testing:

Can be used in situations where there are multiple “correct” outputs, such as user interfaces.
Good tooling will make it possible to quickly compare and update snapshot tests across all tests. This is important if a change to the UI affects most/all tests.
- Good tooling will also make it possible to update multiple snapshot serial assertions in the same test without failing on the first failure (which would require you to re-run the snapshot tool repeatedly until each successive snapshot is updated).
Can be used to determine and capture the behavior of an existing system.
Encourages program design such that components can be driven independently and their state can be dumped/examined in a meaningful way.

Snapshot testing is a better fit for testing TUIs than standard assertions, as much of the interface has functional requirements that aren’t easy to express with assert statements (or similar).

Some snapshot tools store the snapshots inline with the code, while others store them in separate files (or the tool may be configurable). I prefer to store my snapshots inline because I find it easier to reference them when writing or reviewing a test, but this obviously bloats the size of the test file significantly and could make it hard to navigate around the actual test code.

“Expect”-style testing

“Expect”-style testing originally refers to the expect(1) program:

Expect is a program that “talks” to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. An interpreted language provides branching and high-level control structures to direct the dialogue. In addition, the user can take control and interact directly when desired, afterward returning control to the script.

Sometimes “expect”-style testing is used synonymously with “snapshot” testing, such as in Jest or the expecttest Python library.

The basic structure of a program is to send some input, then use expect to wait for certain output. The expect may wait for a simple condition (e.g. wait until the string “Done” appears on the screen) or wait/assert the contents of the entire screen.

End-to-end example

Here’s an example of what an expect-based test might look like for a tool which makes certain Git commits (source, here is the definition of run_pty):

    {
        run_in_pty(
            &git,
            "record",
            &["-i", "-m", "foo"],
            &[
                PtyAction::Write("f"), // expand files
                PtyAction::WaitUntilContains("contents1"),
                PtyAction::Write("q"),
            ],
        )?;
    }

    {
        // The above should not have committed anything.
        let (stdout, _stderr) = git.run(&["show"])?;
        insta::assert_snapshot!(stdout, @r###"
        commit 62fc20d2a290daea0d52bdc2ed2ad4be6491010e
        Author: Testy McTestface <test@example.com>
        Date:   Thu Oct 29 12:34:56 2020 -0100

            create test1.txt

        diff --git a/test1.txt b/test1.txt
        new file mode 100644
        index 0000000..7432a8f
        --- /dev/null
        +++ b/test1.txt
        @@ -0,0 +1 @@
        +test1 contents
        "###);
    }

In fact, the only two primitives for PtyAction are Write and WaitUntilContains. A third option could assert the contents of the entire screen. (Instead, a snapshot test of the output from git show is included.)

Tip: in your testing framework, implement a timeout for expectations and, when the timeout is exceeded, print what the test was expecting and what the screen contents actually were.

Integration example

Here is an example of driving the internal state of a TUI program by passing a list of Events in directly:

#[test]
fn test_toggle_all() -> eyre::Result<()> {
    let before = TestingScreenshot::default();
    let after = TestingScreenshot::default();
    let event_source = EventSource::testing(
        80,
        20,
        [
            Event::ExpandAll,
            before.event(),
            Event::ToggleAll,
            after.event(),
            Event::QuitAccept,
        ],
    );
    let state = example_contents();
    let recorder = Recorder::new(state, event_source);
    recorder.run()?;

    insta::assert_display_snapshot!(before, @r###"
    "[File] [Edit] [Select] [View]                                                   "
    "(~) foo/bar                                                                  (-)"
    "        ⋮                                                                       "
    "       18 this is some text                                                     "
    "       19 this is some text                                                     "
    "       20 this is some text                                                     "
    "  [~] Section 1/1                                                            [-]"
    "    [×] - before text 1                                                         "
    "    [×] - before text 2                                                         "
    "    [×] + after text 1                                                          "
    "    [ ] + after text 2                                                          "
    "       23 this is some trailing text                                            "
    "[×] baz                                                                      [-]"
    "        1 Some leading text 1                                                   "
    "        2 Some leading text 2                                                   "
    "  [×] Section 1/1                                                            [-]"
    "    [×] - before text 1                                                         "
    "    [×] - before text 2                                                         "
    "    [×] + after text 1                                                          "
    "    [×] + after text 2                                                          "
    "###);
    insta::assert_display_snapshot!(after, @r###"
    "[File] [Edit] [Select] [View]                                                   "
    "(~) foo/bar                                                                  (-)"
    "        ⋮                                                                       "
    "       18 this is some text                                                     "
    "       19 this is some text                                                     "
    "       20 this is some text                                                     "
    "  [~] Section 1/1                                                            [-]"
    "    [ ] - before text 1                                                         "
    "    [ ] - before text 2                                                         "
    "    [ ] + after text 1                                                          "
    "    [×] + after text 2                                                          "
    "       23 this is some trailing text                                            "
    "[ ] baz                                                                      [-]"
    "        1 Some leading text 1                                                   "
    "        2 Some leading text 2                                                   "
    "  [ ] Section 1/1                                                            [-]"
    "    [ ] - before text 1                                                         "
    "    [ ] - before text 2                                                         "
    "    [ ] + after text 1                                                          "
    "    [ ] + after text 2                                                          "
    "###);
    Ok(())
}

Notice how high-level events like “expand all” are passed to the TUI application. Implicitly, the TUI application is designed such that the “expand all” event will be fully processed and the interface will be re-rendered by the time we process the next event after.event() and capture the screenshot.

The entire terminal user interface, including components such as a menu bar at the top, is captured. (If the menu bar items change, then the snapshot tests will all need to be updated, which is why good tooling is so important.)

Tip: when comparing against a screen capture, include delimiters before and after the ends of lines (here we use the double quote character ") so that you can more easily compare leading and trailing whitespace, and so that editors don’t automatically strip whitespace or flag it as a warning.

The following are hand-curated posts which you might find interesting.

Date		Title
24 Aug 2023		Writing brittle code
01 Sep 2023		On trivial changes
30 Dec 2023	(this post)	Testing terminal user interface apps
30 Oct 2025		Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Comments

On trivial changes

Fri, 01 Sep 2023 00:00:00 +0000

Intended audience	Software engineers. Anyone with checklists as part of their workflows or routines.
Origin	General software engineering experience.
Mood	Amused.

Table of contents:

Related posts
Comments

One day, perhaps three years into my career, I had a very small, one-line code change to make. I don’t remember exactly what it was, but I think it involved computing a value and interpolating it into a string. I considered committing it directly, but submitted it for code review despite how small of a change it was.

My reviewers pointed out three different bugs, which makes it perhaps the buggiest commit I’ve ever written (by density of bugs per lines of code). Since that day, I always submit code for review and I always wait for builds and tests to finish, even for changes which “obviously” don’t affect anything.

The following are hand-curated posts which you might find interesting.

Date		Title
24 Aug 2023		Writing brittle code
01 Sep 2023	(this post)	On trivial changes
30 Dec 2023		Testing terminal user interface apps
30 Oct 2025		Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Want to see more of my posts? Follow me on Twitter or subscribe via RSS.

Steno & PL

🙏: please or thank you?

Question

Results

Responses

Chat A (n = 3)

Chat B (n = 4)

Chat C (n = 2)

Chat D (n = 2)

Related posts

Comments

Datalog DSL detects defective dependency declarations, defanging dodgy development discipline

Slides

Speaker notes

Related posts

Comments

Half my work is adding a cache

Related posts

Comments

Lithe, less analysis with Datalog

Soufflé

Rapid prototyping

The curse of brevity

Related posts

Comments

Searching source control graphs

Slides

Speaker notes

Related posts

Comments

Language-learning anecdotes

Miscellany

We all scream for ice cream?

On hedgehogs

Social deduction

Forgetting my native language

Sentence formation

Hacking the Latin word ordering

Making my English worse with Latin

Making my English worse with Polish

Preferring a completely-foreign word order

Failing to communicate privately in American sign language

Pronunciation

Phonological production in Polish, but not distinction

Subvocalizing written Chinese

Adjusting my English accent

Making others’ Polish accents worse

Failing to improve my Polish accent

Grammar

Urdu gender confusion

Polish gender confusion

Urdu grammar confusion

Related posts

Comments

Incremental processing with Watchman — don't daemonize that build!

Slides

Speaker notes

Related posts

Comments

Patch terminology

Methodology

Results

Remarks

Conclusions

Related posts

Comments

Testing terminal user interface apps

Introduction

Primer on TTYs and PTYs

Escape codes

Terminal emulation

Testing patterns

Integration vs end-to-end

Snapshot testing

“Expect”-style testing

End-to-end example

Integration example

Related posts

Comments

On trivial changes