Massive TIME_WAIT socket exhaustion during metadata (manifest/avro) reads with S3FileIO + Apache HTTP client

[Sbaia]

Apache Iceberg version

1.10.0

Query engine

Spark

Please describe the bug 🐞

We are seeing severe outbound socket exhaustion (TIME_WAIT) when running Iceberg maintenance operations (specifically CALL system.rewrite_data_files) on large tables stored on S3.

This happens even with Apache HTTP client + connection pooling enabled and after removing any Hadoop/S3A usage.
The issue seems to correlate strongly with metadata / manifest (.avro) downloads, not with large data file reads.

---

Environment

• Iceberg version: 1.10.0
• Spark version: 4.0.1
• Spark on Kubernetes (Spark Operator / SparkApplication CRD)
• Storage: Amazon S3
• FileIO: org.apache.iceberg.aws.s3.S3FileIO
• HTTP client: Apache HTTP client (Iceberg shaded)
• AWS SDK: via iceberg-aws-bundle
• No s3a://, no hadoop-aws in use
• REST Catalog (Lakekeeper), but REST traffic is minimal; sockets are clearly to S3

---

Observed behavior

During rewrite_data_files on a table with ~26k data files:

• Outbound connections to S3 explode to 40k–45k sockets in TIME_WAIT

• Remote IPs are public S3 endpoints (3.x, 52.x)

• Happens primarily while reading metadata files:

  • metadata/*.json
  • manifest-list.avro
  • snap-*.avro

• Kernel ephemeral ports get exhausted, causing job instability

And socket inspection from inside the executor pod:

~43k TIME_WAIT sockets
Top destinations:
- 3.5.x.x
- 52.218.x.x

---

Relevant configuration

spark.sql.catalog.lakehouse.io-impl=org.apache.iceberg.aws.s3.S3FileIO

spark.sql.catalog.lakehouse.http-client.type=apache
spark.sql.catalog.lakehouse.http-client.apache.max-connections=200
spark.sql.catalog.lakehouse.http-client.apache.connection-max-idle-time-ms=300000
spark.sql.catalog.lakehouse.http-client.apache.connection-time-to-live-ms=3600000

spark.sql.iceberg.planning.max-threads=4   # reducing to 1 helps but does not eliminate

---

Why this looks like an Iceberg-level issue

• The connection explosion correlates with manifest/metadata access, not data file I/O
• Planning and rewrite phases appear to trigger bursty, highly parallel small-object GETs
• Even with pooling, connections are frequently closed and recreated

This suggests:

• Metadata access patterns may be too aggressively parallel
• Manifest downloads may bypass or defeat effective connection reuse
• Planning threads / metadata splits may cause connection churn beyond what pooling can absorb

---

Questions / possible directions

• Is metadata/manifest I/O intentionally parallelized at this level?
• Are there known issues with connection reuse during manifest reads?
• Should planning.max-threads or metadata split behavior be auto-throttled?
• Are there additional cache knobs or client reuse guarantees for metadata reads?
• Has similar behavior been observed or addressed in newer versions?

We’re happy to provide:

• Additional logs (with request paths)
• Repro steps
• Packet/socket stats
• A minimal test case if needed

Thanks — this one is pretty brutal in production environments with strict networking limits.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[steveloughran]

• did you initially see this with the s3a code?
• is there a proxy in the way?

only times we've seen TIME_WAIT issues with s3a code is hbase compaction where seek() operations round the fS were triggering abort on the http channel, rather than the clean close you'd expect if a ranged get were used.

If you want to play with the s3a code, set

fs.iostatistics.logging.level=info

this will dump all collected stats on what's happening internally, and would be very informative for a view of what's happening at the bottom.

[eshishki]

can this be a problem 9632a2f?
updated to 4.1 and see a lot of pool timeout errors

what worked for me was
spark.sql.catalog.glue.http-client.apache.max-connections=500
increase from 50 to 500

spark.driver.extraJavaOptions=-Diceberg.worker.delete-num-threads=32
reduce from 4cpu_cores to 1cpu_cores

spark.executor.extraJavaOptions=-Diceberg.worker.delete-num-threads=32
the same but for executor

[steveloughran]

consider opening pool timeout as a separate issue. That is usually (s3a) is caused by not enough http connections for the workers or too many streams/files being kept open with active GET read data coming in

[nastra]

@Sbaia it would be interesting to know whether this is also an issue that you're experiencing with Iceberg 1.9.0 or whether this is something that started to happen with 1.10.0 (where 9632a2f got introduced)

[mkvenkatesh]

@Sbaia it would be interesting to know whether this is also an issue that you're experiencing with Iceberg 1.9.0 or whether this is something that started to happen with 1.10.0 (where 9632a2f got introduced)

That commit was introduced in 1.10.1? https://iceberg.apache.org/releases/#1101-release

AWS: Configure builder for reuse of HTTP connection pool in SDK v2 ([#14161](https://github.com/apache/iceberg/pull/14161))

We are on 1.10.0 and see the problem described in this issue as well. We will switch to 1.10.1 and see if that fixes the problem.

Stack trace on 1.10.0

software.amazon.awssdk.core.exception.SdkClientException: Unable to
  execute HTTP request: Cannot assign requested address
      at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
      at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper2.setLastException(RetryableStageHelper2.java:226)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:65)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
      at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
      at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
      at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
      at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
      at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
      at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
      at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
      at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
      at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:66)
      at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
      at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60)
      at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
      at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
      at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:5848)
      at org.apache.iceberg.aws.s3.S3InputStream.openStream(S3InputStream.java:244)
      at org.apache.iceberg.aws.s3.S3InputStream.openStream(S3InputStream.java:229)
      at org.apache.iceberg.aws.s3.S3InputStream.positionStream(S3InputStream.java:225)
      at org.apache.iceberg.aws.s3.S3InputStream.read(S3InputStream.java:143)
      at com.indeed.shadow.deployable.org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:102)
      at com.indeed.shadow.deployable.org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:126)
      at com.indeed.shadow.deployable.org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91)
      at com.indeed.shadow.deployable.org.apache.parquet.hadoop.ParquetFileReader$ConsecutivePartList.readAll(ParquetFileReader.java:2264)
      at com.indeed.shadow.deployable.org.apache.parquet.hadoop.ParquetFileReader.readAllPartsVectoredOrNormal(ParquetFileReader.java:1283)
      at com.indeed.shadow.deployable.org.apache.parquet.hadoop.ParquetFileReader.internalReadRowGroup(ParquetFileReader.java:1185)
      at com.indeed.shadow.deployable.org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:1135)
      at org.apache.iceberg.parquet.ParquetReader$FileIterator.advance(ParquetReader.java:161)
      at org.apache.iceberg.parquet.ParquetReader$FileIterator.next(ParquetReader.java:130)
      at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:65)
      at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:49)
      at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:135)
      at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:120)
      at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:158)
      at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:63)
      at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:63)
      at scala.Option.exists(Option.scala:376)
      at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$anon$1.hasNext(DataSourceRDD.scala:63)
      at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
      at scala.collection.Iterator$anon$10.hasNext(Iterator.scala:460)
      at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
      at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
      at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
      at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.$anonfun$run$5(WriteToDataSourceV2Exec.scala:446)
      at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1397)
      at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run(WriteToDataSourceV2Exec.scala:491)
      at org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run$(WriteToDataSourceV2Exec.scala:430)
      at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:496)
      at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:393)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
      at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
      at org.apache.spark.scheduler.Task.run(Task.scala:141)
      at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
      at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
      at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:840)
      Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Cannot assign requested address
      Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Cannot assign requested address
      Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: Cannot assign requested address
      Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 4 failure: Unable to execute HTTP request: Cannot assign requested address
      Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 5 failure: Unable to execute HTTP request: Cannot assign requested address
  Caused by: java.net.BindException: Cannot assign requested address
      at java.base/sun.nio.ch.Net.connect0(Native Method)
      at java.base/sun.nio.ch.Net.connect(Net.java:579)
      at java.base/sun.nio.ch.Net.connect(Net.java:568)
      at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593)
      at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
      at java.base/java.net.Socket.connect(Socket.java:639)
      at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
      at software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory.connectSocket(SdkTlsSocketFactory.java:63)
      at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
      at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
      at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
      at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
      at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
      at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
      at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
      at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228)
      at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:102)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
      at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
      at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
      at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
      at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
      ... 66 more

[steveloughran]

The only time I've ever seen BindExceptions was when setting up a socket to listen on a port and the port was in use.

Here maybe there's something up with your local /etc/hosts setting or similar such that httpclient cannot bind to an ephemeral port on what it thinks is your IP address.

I'd be very surprised if this was anything other than client-side config issues.

[cristian-fatu]

We have the same issue, on Spark 3.5.8 and Iceberg 1.10.1. It only seems to happen with tables that have a lot of fragmentation (lots of data files, lots of metadata and lots of position delete files; unclear which one is triggering the problem).
When checking available ports on the Spark executor there are about 40k to begin with. When the error happens later on we see 40K connections to S3 in TIME_WAIT (so all ephemeral ports are used up).
If I compact the table with Trino and later on run compaction with Spark (so there are only a few files as input) everything works fine.

[mkvenkatesh]

The only time I've ever seen BindExceptions was when setting up a socket to listen on a port and the port was in use.

Here maybe there's something up with your local /etc/hosts setting or similar such that httpclient cannot bind to an ephemeral port on what it thinks is your IP address.

I'd be very surprised if this was anything other than client-side config issues.

Thank you. Moving to 1.10.1 didn't improve things so we fiddled with the HTTP client settings and iceberg.worker.delete-num-threads and the BindExceptions have disappeared.

Final settings that helped for us

-- http client configs

http-client.apache.max-connections: 50
http-client.apache.connection-time-to-live-ms: 300000
http-client.apache.connection-max-idle-time-ms: 30000
http-client.apache.tcp-keep-alive-enabled: true

-- spark conf

--conf spark.executor.extraJavaOptions="-Diceberg.worker.delete-num-threads=8"

[steveloughran]

so if it is during delete, you could change the size of each batch delete up from its default of 250 to a value up to 1000.

s3.delete.batch-size 500

that means half as many http connections are needed.
But as each file deleted still counts against your 3500 write/second allocation, large deletes can have adverse consequences, see apache/hadoop@56dee66 for the details.

[mkvenkatesh]

so if it is during delete, you could change the size of each batch delete up from its default of 250 to a value up to 1000.

s3.delete.batch-size 500

that means half as many http connections are needed. But as each file deleted still counts against your 3500 write/second allocation, large deletes can have adverse consequences, see apache/hadoop@56dee66 for the details.

Thank you @steveloughran . The errors we see are from the compaction job and not remove unreferenced files. We are seeing two scenarios where the BindExceptions happened for us

1. When compaction job tries to materialize all the equality deletes within a file group in a Spark job.
2. When we end up having one file group (one spark job) with only one spark task processing > 1K data files (it's possible some of these are equality/positional delete files as well). We have these tiny 14KB data files strewn across multiple s3 prefixes all belonging to one partition, so even with a very low max-file-group-size-bytes we still end up with one spark job/task to process > 1K files and we see BindExceptions around that time.

I don't know which settings helped which case, but our problems disappeared. Increased delete-num-threads to 32 but that caused problems again, so reduced it down to 16 and we see no connection issues.