CEP-0071: Language-Integrated Test Framework¶

Draft

Draft proposal for source-level test declarations and compiler-owned test discovery. V1 accepts only std.testing.assert and does not require language-integrated tests.

Summary¶

Catalyst should eventually support Zig-like source-level test declarations that are discovered, checked, and run by the compiler test command.

This proposal owns the language-integrated part of testing: how tests are declared in source, how they participate in parsing and semantic analysis, how the compiler discovers them, how deterministic test identity and ordering work, and how test failures are reported. std.testing continues to own assertion helpers, equality helpers, fixtures, and test-support APIs.

Motivation¶

Tests are most useful when they live next to the code they exercise and use the same name-resolution, privacy, diagnostics, and comptime machinery as ordinary source. A language-integrated test declaration avoids making test discovery depend on naming conventions, external manifests, or reflection over ordinary functions.

Zig's test "name" { ... } shape is attractive for Catalyst because it is explicit source syntax, cheap to discover, and not part of the production runtime surface. Catalyst should preserve those benefits while keeping its own boundaries clear: test declarations are compiler-recognized declarations, not hidden runtime functions or magical std calls.

Proposed Direction¶

The future design should consider a declaration form similar to:

test "integer addition" {
  const value: i32 = 20 + 22
  std.testing.assert(value == 42)
}

Test declarations would be accepted only where the language explicitly allows them, likely at file or namespace declaration scope. They would not export ordinary callable symbols, participate in normal overload sets, or run during production compilation.

The compiler test command should discover test declarations through parsed source artifacts, semantically check their bodies with the same rules as ordinary code, lower accepted test bodies through the ordinary pipeline, and execute them through the selected test backend. Test declarations should have deterministic identities derived from module identity, source location, and explicit test name where present.

The design should cover:

whether test is a reserved word, contextual declaration starter, or attribute-driven declaration form;
accepted declaration positions and visibility behavior;
whether tests may access private declarations in their containing file or module;
test body result types, including void, void!E, and try-based failure;
deterministic discovery and execution order;
filtering by test name, module, package, or source path;
how assertion traps, returned errors, compiler diagnostics, panics, and resource leaks are reported as test failures;
how source spans and notes are presented in test output;
whether skipped, expected-failing, or expected-diagnostic tests are language syntax, std.testing APIs, or runner metadata;
interaction with comptime, attributes, conditional declarations, modules, imports, and include files;
whether test declarations can carry metadata attributes;
whether test-only declarations or imports exist, and how they avoid leaking into production builds;
how generated C, interpreter runs, and future native backends expose the same test contract.

Example¶

Possible future shape:

test "fixed buffer allocator reports exhaustion" {
  var storage: [64]u8 = undefined
  var allocator = std.mem.FixedBufferAllocator.init(&storage)

  const first = try allocator.alloc(u8, 64)
  std.testing.assert(first.len == 64)

  const second = allocator.alloc(u8, 1) catch as err {
    std.testing.assert(err == .OutOfMemory)
    return void
  }

  _ = second
  std.testing.assert(false, "allocation should have failed")
}

This is illustrative future syntax. V1 does not accept test declarations as source syntax.

Failure Model¶

A test fails when its body emits a hard diagnostic during checking, returns or propagates an uncaught error according to the accepted test-body result model, triggers a checked trap such as a failed std.testing.assert, violates a test-runner resource policy, or times out under a future explicit runner policy.

The failure report should be deterministic. It should avoid pointer addresses, host-specific paths, nondeterministic ordering, and hidden dependencies on terminal color or platform formatting.

V1 Compatibility¶

V1 does not require language-integrated test declarations or a compiler test command for Catalyst source. V1 accepts std.testing.assert(condition: bool, message: ?[]const u8 = null) as a standard helper that can be used by examples, implementation tests, and future test declarations.

Until this CEP is accepted, compiler implementation tests should use the implementation's external test harness, snapshot fixtures, and host-language tests rather than treating test as Catalyst syntax.