Could you please explain if there will be spilling in native mode?

[shentaoss]

When I run it in native mode, it easily triggers OOM, but on Ray, a single node doesn't encounter this issue.

[AkhilSirvi]

Daft can hit OOM in native mode because everything has to fit in memory. With Ray, Daft benefits from Ray’s object spilling so data can go to disk under memory pressure, which is why you don’t see the same issue there.
Docs mention memory pressure + spill behavior here:
https://docs.daft.ai/en/stable/optimization/memory/

[shentaoss]

Got it, thanks.