Parameterizing TypedArrays

#58573

• We didn't get the chance to add the es2024 target

• ArrayBuffer got new members that SharedArrayBuffer does not have.

• Previously, SharedArrayBuffer just had two members apart from byteLength and slice, making them interchangeable.

• es2024 has new members for each of these.

  • Proposed changes make them no longer interchangeable.
  • ArrayBufferLike is the best type to describe both.

• Why?

  • The WebCrypto APIs only allow ArrayBuffer, and not SharedArrayBuffer,
    • e.g. crypto.subtle.digest
  • Also, ArrayBuffer is not a transferable object.

• Makes it hard when you try to get the underlying buffer via someUint8Array.buffer.

• What is the underlying idea of how these compose?

  • ArrayBuffer is a non-indexable span of memory. You use an "ArrayBuffer view" to access the memory.
    • They're not thread-safe. They're only meant to be read/written from within a single thread. If you want to share memory, you either copy the memory or transfer it entirely.
  • SharedArrayBuffer looks like an ArrayBuffer but operates over memory in a shared global heap and has has unordered but sequentially consistent writes.

• So the idea is to parameterize each of these views over the underlying buffer type.

  interface Uint8Array<Buffer extends ArrayBufferLike = ArrayBufferLike> {
    // ...
    readonly buffer: Buffer;
  
  
    // Most methods and constructors return a view with a local-only ArrayBuffer
    new (length: number): Uint8Array<ArrayBuffer>;
  
    // (not this one)
    new <T extends ArrayBufferLike>(buffer: T, byteOffset?: number, length?: number): Uint8Array<T>;
    
    new (array: ArrayLike<number> | ArrayBuffer): Uint8Array<ArrayBuffer>;
    // ...
    filter(predicate: /*...*/): Uint8Array<ArrayBuffer>;
  }

  • Note the above code is roughly transcribed, don't look at this as precise.

• Problems:

  • Buffer subtypes Uint8Array.
    • We say if you extend a base type, that base type has to have a consistent construct signature.
    • Would have to make Buffer generic - and to do that, we would have to start using typesVersions because old UInt8Array aren't generic.
      • Workaround: just change the returned type to Buffer & WithArrayBufferLike<...> in the retun types of slice and subarray.
      • Why not just forward-declare UInt8Array as generic with an option type parameter?
  • Also, return this in some cases.

• What if we fixed up stuff like crypto.subtle.digest etc. to accept SharedArrayBuffer even though they don't take those?

  • Fixes the DOM, but doesn't fix everything.

• Could say the underlying default should be ArrayBuffer, not ArrayBufferLike.

  • We created ArrayBufferLike and traditionally these have never had a noticeable difference.

File Extension Rewriting, --experimental-transform-types/--experimental-strip-types, and Multi-Project Builds

#59767

• Last week, we discussed rewriting relative file extensions. Had concerns, mainly around monorepo-style codebases.

• In the meantime, we have a prototype PR.

• Sample project

  // packages/lib/src/math.ts
  export function add(a: number, b: number) {
    return a + b;
  }
  
  // packages/lib/src/main.ts
  export * from "./math.ts";
  
  // packages/app/src/main.ts
  import { add } from "@typescript-node/lib";
  
  console.log(add(1, 2));

• By default doesn't, work, but...

  {
    // ...
  
    "exports": {
        ".": {
            "typescript": "./src/main.ts",
            "import": "./dist/main.js",
        }
    }
  }

  • Works when you run with node --conditions typescript.

• Almost right, but it's not safe to publish TypeScript - if this exports map was published to npm and run with node --conditions typescript, resolution would fail within the published package.

• One way is to erase here - but no built-in tooling to do this.

• @colinhacks suggested namespacing on a per-package basis for publishing.

  {
    // ...
  
    "exports": {
        ".": {
            "@my-special-namespace/source": "./src/main.ts",
            "import": "./dist/main.js",
        }
    }
  }

  • Can also erase these, but not sure what tools do that.

• moduleSuffixes

  • Nothing special needed there, but you can't really take advantage of extension rewriting in certain circumstances.
    • You can't name something foo.ts.android.ts, but you also can't write foo.ts.ts anyway.
    • Probably will be very rare - this is mainly for React Native, and frankly really unhinged to do this.

• Now what if projects don't take advantage of workspaces and just do a direct relative import?

• import { add as _add } from "../../lib/src/main.ts

  • Won't work if outDir is dist because it needs to be rewritten to ../../lib/dist/main.ts.
  • It just doesn't work in some circumstances and we can give an error there.

• You still can use relative imports - everything just needs to end up in the same output folder. This is, for example, how TypeScript's build works! So even we could do this.

• For clarity: relative imports work for the following...

  root/
  ├── src/
  │   ├── projA/
  │   ├── projB/
  │   └── projC/
  └── dist/
      ├── projA/
      ├── projB/
      └── projC/
  

  but relative imports do not work for the following.

  projects/
  ├── projA/
  │   ├── src/
  │   └── dist/
  ├── projB/
  │   ├── src/
  │   └── dist/
  └── projC/
      ├── src/
      └── dist/
  

• Sample PR to arethetypeswrong that makes everything work with --experimental-transform-types: Use --experimental-transform-types arethetypeswrong/arethetypeswrong.github.io#194

  • Notable interesting details:
    • Tests that can work against both the TS source and JS output! One just passes a specific --conditions.
    • Thought you needed tsconfig custom conditions- but you don't. TypeScript's project references are smart enough to map output files to input files.
    • Did a regex replace on relative paths - got one wrong in #internal/getProbableExports.js to #internal/getProbableExports.
      • Reinforced the need for good errors.

• What would all this look like without project references?

  • Like one big tsconfig.json?
    • Not necessarily.
  • Probably works, just need to break things apart by packages and can't use relative paths.

• Should Node automatically have a condition?

  • Interesting long-term, but maybe it's good for people to have a specific level of control.

• Boilerplate-y to have to write --conditions @my-namespace/source and @my-namespace/source throughout exports/imports, but probably worth the control.

  • It's not just boilerplate though, it's more about not having all these conditions published, and not exposing this to users. Really would be ideal if these conditions could be automatically erased before publishing.

• otherwise, we feel good about this. It's not 0-config throughout, but it feels like there is a story between the "I can start a Node server fast" and "I can break my projects apart into multiple pieces"/"I want to publish stuff to npm" that we feel good about.