Apache Iceberg version
1.10.0
Query engine
Spark
Please describe the bug 馃悶
We observed this with Iceberg views through a REST catalog backed by Apache Polaris.
ViewMetadata.Builder.build() correctly limits retained view versions using version.history.num-entries, but schemas referenced only by expired versions remain in schemas.
For example, with history size 2:
v1 -> schema 1
v2 -> schema 2
v3 -> schema 3
v4 -> schema 4
The retained versions may correctly be:
v3 -> schema 3
v4 -> schema 4
while the current metadata still contains schemas 1, 2, 3, 4.
For frequently replaced views, this causes the current metadata JSON to grow continuously even though version history is bounded.
I think schemas should be retained based on reachability from the retained ViewVersions. After version retention, only schemas referenced by retained versions should remain. This also preserves shared schemas and the current version's schema naturally.
Willingness to contribute
Apache Iceberg version
1.10.0
Query engine
Spark
Please describe the bug 馃悶
We observed this with Iceberg views through a REST catalog backed by Apache Polaris.
ViewMetadata.Builder.build()correctly limits retained view versions usingversion.history.num-entries, but schemas referenced only by expired versions remain inschemas.For example, with history size
2:The retained versions may correctly be:
while the current metadata still contains schemas
1, 2, 3, 4.For frequently replaced views, this causes the current metadata JSON to grow continuously even though version history is bounded.
I think schemas should be retained based on reachability from the retained
ViewVersions. After version retention, only schemas referenced by retained versions should remain. This also preserves shared schemas and the current version's schema naturally.Willingness to contribute