Describe the bug
Five date-part extraction functions in NativeConverters.scala use buildExtScalarFunction, which does not pass the session timezone to the native Rust implementation:
By contrast, Hour, Minute, Second, and WeekOfYear correctly use buildTimePartExt, which passes sessionLocalTimeZone for TimestampType inputs.
This inconsistency can cause incorrect results for timestamp inputs near date boundaries in non-UTC timezones.
Affected functions:
- Year (Spark_Year) — not timezone-aware
- Month (Spark_Month) — not timezone-aware
- DayOfMonth (Spark_Day) — not timezone-aware
- DayOfWeek (Spark_DayOfWeek) — not timezone-aware
- Quarter (Spark_Quarter) — not timezone-aware
To Reproduce
- Set
spark.sql.session.timeZone to America/New_York
- Create a table with a timestamp column containing
2021-01-04 04:30:00 UTC
(equivalent to 2021-01-03 23:30:00 in New York)
- Run:
SELECT dayofmonth(ts), dayofweek(ts) FROM t1
Expected behavior
All date-part extraction functions should interpret timestamp inputs in the session local timezone before extracting the date component, matching Spark's behavior.
All 5 functions should use buildTimePartExt (or equivalent), and their corresponding Rust implementations should accept and handle an optional timezone argument.
Screenshots
Additional context
Describe the bug
Five date-part extraction functions in
NativeConverters.scalausebuildExtScalarFunction, which does not pass the session timezone to the native Rust implementation:By contrast,
Hour,Minute,Second, andWeekOfYearcorrectly usebuildTimePartExt, which passessessionLocalTimeZoneforTimestampTypeinputs.This inconsistency can cause incorrect results for timestamp inputs near date boundaries in non-UTC timezones.
Affected functions:
To Reproduce
spark.sql.session.timeZonetoAmerica/New_York2021-01-04 04:30:00 UTC(equivalent to
2021-01-03 23:30:00in New York)Expected behavior
All date-part extraction functions should interpret timestamp inputs in the session local timezone before extracting the date component, matching Spark's behavior.
All 5 functions should use
buildTimePartExt(or equivalent), and their corresponding Rust implementations should accept and handle an optional timezone argument.Screenshots
Additional context