[SPARK-55790][Geo][SQL] Build a complete SRS registry using PROJ 9.7.1 data#54571
[SPARK-55790][Geo][SQL] Build a complete SRS registry using PROJ 9.7.1 data#54571uros-db wants to merge 2 commits intoapache:masterfrom
Conversation
uros-db
left a comment
uros-db
left a comment
There was a problem hiding this comment.
@szehon-ho @cloud-fan Please review.
uros-db
left a comment
uros-db
left a comment
There was a problem hiding this comment.
Docker integration failure seems unrelated to these changes.
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
- the pr has two generatedcsv which is very big. just wondering, is there no way to have one generated one, and copy it at build time?
- do we plan to run it just manually? I guess proj does not release that frequently that its worth to automate it?
| @@ -0,0 +1,377 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
should we use hyphen delimiter for this script name, like other dev scripts?
| url = PROJ_RAW_URL.format(version=version, filename=filename) | ||
| print(f" Downloading {url}") | ||
| try: | ||
| with urllib.request.urlopen(url) as response: |
There was a problem hiding this comment.
do we want to add an overridable timeout, to not hang the script (if we ever call it from a build job, for example)
| 'geocentric', 'other' -> non-geographic | ||
|
|
||
| Returns a list of (srid, string_id, is_geographic) tuples, | ||
| excluding deprecated entries and entries with non-numeric codes. |
There was a problem hiding this comment.
is it right? I dont see the script filtering deprecated entries?
| try: | ||
| with urllib.request.urlopen(url) as response: | ||
| return response.read().decode("utf-8") | ||
| except urllib.error.HTTPError as e: |
There was a problem hiding this comment.
should we catch other error like, URLError?
|
|
||
| print() | ||
|
|
||
| # Deduplicate: keep the first occurrence of each SRID. |
There was a problem hiding this comment.
does this happen? should we comment which gives precedence (ESPG/ESRI)
| if not match: | ||
| continue | ||
| fields = parse_sql_values(match.group(1)) | ||
| auth_name = fields[0] |
There was a problem hiding this comment.
nit: add a check for fields length
2 participants
What changes were proposed in this pull request?
Build a more complete Spatial Reference System (SRS) registry in Spark, introducing 10000+ entries sourced from the PROJ library's EPSG and ESRI databases, which will unlock substantially improving the breadth of geospatial type support.
Why are the changes needed?
This will enable Geometry and Geography types to support 10000+ additional SRID/CRS values, both in JVM and Python. Note that currently Geometry and Geography types offer only limited SRID support (a few hardcoded values).
Does this PR introduce any user-facing change?
No.
How was this patch tested?
The script and auto-generated golden files are self-contained.
Was this patch authored or co-authored using generative AI tooling?
Yes, Claude 4.6 Opus.