【error】SparkUI端口被占用

最新推荐文章于 2024-07-15 17:58:41 发布

原创

最新推荐文章于 2024-07-15 17:58:41 发布 · 1.9k 阅读

收录于

当Spark任务启动时，SparkUI默认使用4040端口，若被占用会尝试递增端口，最多重试16次。超过次数会导致任务失败。解决办法是在创建SparkConf时设置`spark.port.maxRetries`参数为更高值，如100，以增加重试连接端口的次数。

 ERROR ui.SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: bind: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:317)
	at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
	at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.spark.ui.JettyU