[#8725] feat(core): Support downloading job artifacts from hcfs #8813
Conversation
| // We need to add more later on when we have more catalog implementations. | ||
| return barrierClasses.stream().anyMatch(name::startsWith); | ||
| return barrierClasses.stream().anyMatch(name::startsWith) | ||
| || name.startsWith("org.apache.hadoop"); |
There was a problem hiding this comment.
Why not just put "org.apache.hadoop" in the barrierClasses, Is the barrierClasses designed to handle this case?
There was a problem hiding this comment.
I'm just testing the code. I will polish it when everything is ready. There still have some other issues according to the CI, I need to investigate more.
There was a problem hiding this comment.
The previous CI is passed, but it introduces other problems, I need to check it.
| put the `hadoop-aws` and its dependencies in the `{GRAVITINO_HOME}/libs` directory. By | ||
| default, the HDFS client libraries are already included in the Gravitino server package. | ||
| 2. The Gravitino server has the corresponding Hadoop configurations in its classpath | ||
| `{GRAVITINO_HOME}/conf` directory. For example, if you want to use `s3a://` scheme, you need to |
There was a problem hiding this comment.
The Gravitino server has the corresponding Hadoop configurations in its classpath
{GRAVITINO_HOME}/confdirectory
I can't get this sentence, as there are no *.xml in the folder {GRAVITINO_HOME}/conf, why do you say that?
Should the word has in Gravitino server has the corresponding be needs?
2 participants
What changes were proposed in this pull request?
This PR adds the support of downloading job artifacts from the Hadoop compatible filesystems.
Why are the changes needed?
Fix: #8725
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Local tests.