82,627 questions
0
votes
1
answer
50
views
Why 2 tables bucketed by col1 and joined by (col1, col2) are shuffled?
// Enable all bucketing optimizations
spark.conf.set("spark.sql.requireAllClusterKeysForDistribution", "false")
spark.conf.set("spark.sql.sources.bucketing.enabled&...
- 31
2
votes
0
answers
59
views
'JavaPackage' object is not callable error when trying to getOrCreate() local spark session
I have set up a small Xubuntu machine with intention of making it my single-node playaround Spark cluster. The cluster seems to be set up correctly - I can access the WebUI at port 8080, it shows a ...
- 558
0
votes
0
answers
17
views
EMR Spark cluster getting stuck on resizing
I have a EMR spark cluster, on which I have enabled EMR managed auto scaling as
auto scaling configuration
and primary - c5a.xlarge
Core - c5a.xlarge
Task - c5a.xlarge
With these cluster ...
- 1
0
votes
1
answer
101
views
Optimize code to flatten meta ads metrics data in spark
I have two spark scripts, first as a bronze script need to data form kafka topics each topic have ads platform data ( tiktok_insights, meta_insights, google_insights ). Structure are same,
( id, ...
- 21
0
votes
0
answers
70
views
spark flatMapToPair reaching "no space left on device" due to large duplication of entries
First, my question is not on increasing disk space to avoid no space left error, but to understand what spark does, and hopefully how to improve my code.
In short, here is the pseudo code:
JavaRDD&...
- 15.8k
1
vote
2
answers
101
views
Difference between org.apache.hadoop.io.compress.CompressionCodec and org.apache.spark.io.CompressionCodec
I want to use a compression in bigdata processing, but there are two compression codecs.
Anyone know the difference?
- 1,150
Advice
0
votes
4
replies
83
views
Use RSA key snowflake connection options instead of Password
I want to connect to a Snowflake database from the Data Bricks notebook. I have an RSA key(.pem file) and I don't want to use a traditional method like username and password as it is not as secure as ...
- 85
0
votes
1
answer
110
views
Does Databricks Spark SQL evaluate all CASE branches for UDFs?
I'm using Databricks SQL and have SQL UDFs for GeoIP / ISP lookups.
Each UDF branches on IPv4 vs IPv6 using a CASE expression like:
CASE
WHEN ip_address LIKE '%:%:%' THEN -- IPv6 path
...
...
- 3
1
vote
0
answers
120
views
Warning and performance issues when scanning delta tables
Why do I get multiple warnings WARN delta_kernel::engine::default::json] read_json receiver end of channel dropped before sending completed when scanning (pl.scan_delta(temp_path) a delta table that ...
- 6,048
1
vote
1
answer
52
views
How to detect Spark application failure in SparkListener when no jobs are executed?
I have a class that extends SparkListener and has access to SparkContext. I'm wondering if there is any way to check in onApplicationEnd whether the Spark application stopped because of an error or ...
- 11
0
votes
0
answers
39
views
How to dynamically cast columns in a dbt-spark custom materialization to resolve UNION ALL schema mismatch?
I am working on a custom materialization in dbt using the dbt-spark adapter (writing to Delta tables on S3). The goal is to handle a hybrid SCD Type 1 and Type 2 strategy.
The Logic I compare the ...
2
votes
0
answers
64
views
How log model in mlflow using Spark Connect
I have the following setup:
Kubernetes cluster with Spark Connect 4.0.1 and
MLflow tracking server 3.5.0
MLFlow tracking server should serve all artifacts and is configured this way:
--backend-store-...
- 6,213
0
votes
1
answer
74
views
Handle corrupted files in spark load()
I have a spark job that runs daily to load data from S3.
These data are composed of thousands of gzip files. However, in some cases, there is one or two corrupted files in S3, and it causes the whole ...
- 1,933
-1
votes
2
answers
63
views
Connectivity issues in standalone Spark 4.0
In Azure VM, I have installed standalone Spark 4.0. On the same VM I have Python 3.11 with Jupyter deployed. In my notebook I submitted the following program:
from pyspark.sql import SparkSession
...
- 43
1
vote
1
answer
144
views
PicklingError: Could not serialize object: RecursionError in pyspark code in Jupyter Notebook
I am very new in Spark (specifically, have just started with learning), and I have encountered a recursion error in a very simple code.
Background:
Spark Version 3.5.7
Java Version 11.0.29 (Eclipse ...
- 11