Serverless is a contract with an allow-list
Databricks serverless refused three of my classic idioms. The best error had an empty string where the type name should be — at table 21 of 43, with twenty Delta targets already created.
The €100 Lakehouse — 6/12
Databricks serverless refused three of my classic idioms. The best error had an empty string where the type name should be — at table 21 of 43, with twenty Delta targets already created.
Serverless is the deliberate choice behind the €100/mo ceiling (auto_stop_mins=1). It is also a contract with an allow-list — and I paid for each clause in hours.
→ spark._jvm, the normal PySpark way to touch the FileSystem: refused with [JVM_ATTRIBUTE_NOT_SUPPORTED]. The error's suggested fix is a single-user cluster — pay more to keep a private idiom. The actual fix: dbutils.fs.
→ Postgres has a `time` type. Spark doesn't. The parquet carried TIME(MICROS) and the job died with "data type is not supported" — type name blank. Fix: cast to TEXT in the extraction SELECT; '09:30:00' sorts and reads fine.
→ psycopg[binary]>=3.2 died with SIGABRT (exit 134) inside the C loader. The serverless image ships its own psycopg; the version range accepted it, and only psycopg-binary was installed fresh — pure-Python package and compiled shim from different releases. Fix: pin both to 3.3.4, same as uv.lock.
Cheap serverless is a contract with an allow-list — every refused idiom has a supported equivalent, and "upgrade a tier" is almost always the wrong answer to a price.
What's the most expensive error message you've ever debugged — and did it even name the thing that was broken?
Databricks · dbt · Airflow · Terraform · AWS
P.S. New tech post every Wednesday.
#100EuroLakehouse #Databricks #Serverless