Issues with include/exclude in Deno

This post is an informal extension of the issue I've filed: denoland/deno_config#169

How it started ​

I stumbled upon an issue. Consider a workspace with the following structure:

deno.json
foo.ts
sub/
  deno.json
  bar.ts

If you call deno fmt (or deno lint, deno test, deno bench), it formats 4 files:

• deno.json and foo.ts with fmt options in the root deno.json;
• sub/deno.json and sub/bar.ts with fmt options from sub/deno.json.

However, if you call deno fmt ., it will format 6 files instead:

• Files matched by the root deno.json workspace member:
  • deno.json
  • foo.ts
• Files matched by the sub/deno.json workspace member:
  • sub/deno.json
  • sub/bar.ts
  • deno.json
  • foo.ts

This is clearly unxpected behaviour (filed: deno#29057).

Why? ​

Because of how Deno handles includes/excludes and paths added in the CLI. In short, it is a mess of ad hoc logic.

Here are some snippets from deno_config:

struct FilePatterns {
  base: PathBuf,
  include: Option<PathOrPatternSet>,
  exclude: PathOrPatternSet
}

struct PathOrPatternSet(Vec<PathOrPattern>);

enum PathOrPattern {
  Path(PathBuf),
  NegatedPath(PathBuf),
  RemoteUrl(Url),
  Pattern(GlobPattern),
}

This directly models include and exclude sets from Deno configuration, like when you write

{
  "exclude": ["./internal"],
  "fmt": {
    "include": ["src/**/*", "!src/generated/"],
    "exclude": ["src/dist"]
  }
}

The thing is: when you add an additional files to a command, like deno fmt [files].., it simply overwrites includes for every workspace member (workspace/mod.rs). In the example above with deno fmt . (or deno fmt <root>), it works this way:

• Files for deno.json: base <root>, include <root>, exclude sub/
• Files for sub/deno.json: base sub/, include <root>, exclude nothing

That it why sub/deno.json also applies to the root directory and not only to the contents of sub/.

Paths in CLI should refine, not extend ​

First, I think that CLI paths should refine the existing include/exclude sets, not extend them. Otherwise there are some difficult questions:

What if I specify a file that is outside of the workspace? E.g. deno fmt ../test.ts. What configuration options should Deno apply then? Using the workspace member I am executing the command from? Using default config?

Instead, a more consistent behaviour would be that if I call deno fmt src foo.ts, it will be the same as I call deno fmt, but limited to the src directory and foo.ts file. Currently it doesn't work this way.

Rewrite using set theory composition ​

I am thinking about redefining FilePatterns from scratch using a simple and composable logic:

struct FilePatterns {
  base: PathBuf,
  matcher: PathsMatcher
}

enum PathOrPatternSet {
  Any(Vec<PathOrPattern>),
  All(Vec<PathOrPatternSet>),
  Not(Box<PathOrPatternSet>)
}

Here:

• Any is an $OR$ (union), and what is now written in include/exclude. It is a list of paths/patterns/negated paths/patterns which work together as a union: if a path is matched by any of them, we consider it to be matching the whole set. However, the complexity arises with negated paths and patterns. As an idea: if we meet a negated path/pattern, it introduces a gap within the union set.
• All is a $AND$ (intersection) of all inner sets matching a path.
• Not is $NOT$ (inversion) of the inner set.

With this, the include/exclude set could be represented as

All(
  Any([include paths]..),
  Not(
    Any([exclude paths]..)
  )
)

or

Then:

• Additional files in CLI is Any([cli paths...]), $AND$-ed to (intersected with) the main set.
• .gitignore paths are wrapped into Not(Any(..)) and $AND$-ed to the main set too, further restricting it.

Issue: current un-exclude behaviour will break ​

Sometimes you may find that you want to un-exclude a path or pattern that's excluded in the top level-exclude. In Deno 1.41.2+, you may un-exclude a path by specifying a negated glob in a more specific config:

{
  "fmt": {
    "exclude": [
      // format the dist folder even though it's
      // excluded at the top level
      "!dist"
    ]
  },
  "exclude": [
    "dist/"
  ]
}

— Deno - Configuration - Top-level exclude

Well, I don't think it should work this way. This approach leads to confusion and inconsistency.

I think it should be that a negated glob makes a gap, a hole in the existing set, and it should not affect anything else:

{
  "exclude": [
    "dist/", // exclude dist
    "!dist/foo", // but not exclude dist/foo
    "!src/something" // doesn't affect anything
  ]
}

Also: collisions ​

Apparently, Deno doesn't check for collisions between workspace members' include patterns. What if two members include the same files in their fmt while specifying different formatting options?

Unresolved questions ​

• How should files be split by base path?
• How should top-level excludes interact with local include/exclude precedence?