File sharing between server / worker computers.

[thiskappaisgrey]

Hi, sorry if this is a dumb question, but if I want to set up hyperqueue to manage a set of worker computers and I want them to share files between the worker computers - do I need to set it up where I have a shared / distributed filesystem that each worker computer needs to mount or figure something out with object storage for each task? To my understanding - when I do hq submit - it just connects to the server defined in server-dir and sends the command that needs to be executed right? I'm assuming if you are trying to run hq submit from a computer that isn't running the hq server, you'd need to copy access.json to that computer. Does hyperqueue copy any files from the computer that runs hq submit to the worker computers? Thanks.

Example, if I had like,

# hello.py
print("hello world")

And I wanted to do

hq submit python hello.py

Would I have to figure out a way to share hello.py with the workers?

[Kobzol]

Hi, no question is dumb :) HyperQueue is indeed primarily focused on systems with shared network filesystems, i.e. HPC clusters. However, it's definitely possible to use it without a shared filesystem (it's just more convenient if you have one). You can find a guide on how to use HQ without a shared filesystem in the documentation.

To answer your questions specifically, yes, when you do hq submit, you send the tasks to be computed to the server, and the server then later distributes them to workers. And yes, if you're submitting from a different computer (without a shared filesystem with the server), you need to copy the access.json file (see documentation for more information).

Does hyperqueue copy any files from the computer that runs hq submit to the worker computers?

No, it does not copy any files. If you need to provide input files to tasks, they need to be available from worker nodes (so if you need to copy them, you must copy them yourself, at least currently).

[thiskappaisgrey]

So, I'm assuming the current working directory gets copied to the server & the workers as well when I submit tasks? If I have a hello.py file in my cwd, that gets copied when I do hq submit right?

[Kobzol]

No, it doesn't :) As I said before, nothing gets copied to the server from the client by itself. If you don't have a shared filesystem, and you need to access any file from the worker, you have to copy them yourself.

[thiskappaisgrey]

Thanks!

[spirali]

For completeness, you can attach a binary data to your task. The only option right now from CLI is to use "--stdin" (https://it4innovations.github.io/hyperqueue/stable/jobs/jobs/).
So you can create a tarball of your current directory, and submit it like

hq submit --stdin my-task-script.sh < my_tarball.tar.gz

and unpack your data in my-task-script.sh from stdin

Note: In this way, the data will be also stored in server as they are part of the of the task definition and if they are big, the server may run out of memory.