spark报错：ERROR SparkUI: Failed to bind SparkUI，解决办法

x_lock

11443人浏览 · 2017-05-21 14:10:58

x_lock · 2017-05-21 14:10:58 发布

错误原因：

每一个Spark任务都会占用一个SparkUI端口，默认为4040，如果被占用则依次递增端口重试。但是有个默认重试次数，为16次。16次重试都失败后，会放弃该任务的运行。
参考：
http://blog.csdn.net/a921122/article/details/45095845

解决办法

根据提示，可以在命令行通过配置，增加尝试的次数，具体来说：
/data/hadoop/spark-1.6.0/bin/spark-shell –conf spark.port.maxRetries=100

参考：
http://blog.javachen.com/2015/06/07/spark-configuration.html?utm_source=tuicool&utm_medium=referral

报错信息

17/05/17 11:05:45 ERROR SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service ‘SparkUI’ failed after 16 retries! Consider explicitly setting the appropriate port for the service ‘SparkUI’ (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils .org <script type="math/tex" id="MathJax-Element-819">.org</script>apache spark <script type="math/tex" id="MathJax-Element-820">spark</script>ui JettyUtils <script type="math/tex" id="MathJax-Element-821">JettyUtils</script> connect <script type="math/tex" id="MathJax-Element-822">connect</script>1(JettyUtils.scala:252)
at org.apache.spark.ui.JettyUtils

a n o n f u n $ 5. a p p l y (J e t t y U t i l s . s c a l a : 262) a t o r g . a p a c h e . s p a r k . u i . J e t t y U t i l s

5.apply(JettyUtils.scala:262)atorg.apache.spark.util.Utils <script type="math/tex" id="MathJax-Element-824">5.apply(JettyUtils.scala:262) at org.apache.spark.util.Utils</script>

anonfun <script type="math/tex" id="MathJax-Element-825">anonfun</script>startServiceOnPort

1.apply <script type="math/tex" id="MathJax-Element-826">1.apply</script>mcVI

sp(Utils.scala:2062)atscala.collection.immutable.Range.foreach <script type="math/tex" id="MathJax-Element-827">sp(Utils.scala:2062) at scala.collection.immutable.Range.foreach</script>mVc

sp(Range.scala:141)atorg.apache.spark.util.Utils <script type="math/tex" id="MathJax-Element-828">sp(Range.scala:141) at org.apache.spark.util.Utils</script>.startServiceOnPort(Utils.scala:2053) at org.apache.spark.ui.JettyUtils

.startJettyServer(JettyUtils.scala:262)atorg.apache.spark.ui.WebUI.bind(WebUI.scala:136)atorg.apache.spark.SparkContext <script type="math/tex" id="MathJax-Element-829">.startJettyServer(JettyUtils.scala:262) at org.apache.spark.ui.WebUI.bind(WebUI.scala:136) at org.apache.spark.SparkContext</script>

anonfun <script type="math/tex" id="MathJax-Element-830">anonfun</script>13.apply(SparkContext.scala:481)
at org.apache.spark.SparkContext

a n o n f u n $ 13. a p p l y (S p a r k C o n t e x t . s c a l a : 481) a t s c a l a . O p t i o n . f o r e a c h (O p t i o n . s c a l a : 236) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . < i n i t > (S p a r k C o n t e x t . s c a l a : 481) a t o r g . a p a c h e . s p a r k . r e p l . S p a r k I L o o p . c r e a t e S p a r k C o n t e x t (S p a r k I L o o p . s c a l a : 1017) a t $ l i n e 3. $ r e a d

i w C . < i n i t > (< c o n s o l e > : 15) a t $ l i n e 3. $ r e a d

line3. <script type="math/tex" id="MathJax-Element-833">line3.</script>read.(:26)
at

line3. <script type="math/tex" id="MathJax-Element-834">line3.</script>read

.(:30)at <script type="math/tex" id="MathJax-Element-835">.(:30) at </script>line3.

read <script type="math/tex" id="MathJax-Element-836">read</script>.()
at

line3. <script type="math/tex" id="MathJax-Element-837">line3.</script>eval

.(:7)at <script type="math/tex" id="MathJax-Element-838">.(:7) at </script>line3.

eval <script type="math/tex" id="MathJax-Element-839">eval</script>.()
at

line3. <script type="math/tex" id="MathJax-Element-840">line3.</script>eval.

print()atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)atjava.lang.reflect.Method.invoke(Method.java:498)atorg.apache.spark.repl.SparkIMain <script type="math/tex" id="MathJax-Element-841">print() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.repl.SparkIMain</script>ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain

Request.loadAndRun(SparkIMain.scala:1346)atorg.apache.spark.repl.SparkIMain.loadAndRunReq <script type="math/tex" id="MathJax-Element-842">Request.loadAndRun(SparkIMain.scala:1346) at org.apache.spark.repl.SparkIMain.loadAndRunReq</script>1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret

1(SparkILoop.scala:857)atorg.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)atorg.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)atorg.apache.spark.repl.SparkILoopInit <script type="math/tex" id="MathJax-Element-843">1(SparkILoop.scala:857) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) at org.apache.spark.repl.SparkILoopInit</script>

anonfun <script type="math/tex" id="MathJax-Element-844">anonfun</script>initializeSpark

1.apply(SparkILoopInit.scala:125)atorg.apache.spark.repl.SparkILoopInit <script type="math/tex" id="MathJax-Element-845">1.apply(SparkILoopInit.scala:125) at org.apache.spark.repl.SparkILoopInit</script>

anonfun <script type="math/tex" id="MathJax-Element-846">anonfun</script>initializeSpark

1.apply(SparkILoopInit.scala:124)atorg.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)atorg.apache.spark.repl.SparkILoopInit <script type="math/tex" id="MathJax-Element-847">1.apply(SparkILoopInit.scala:124) at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) at org.apache.spark.repl.SparkILoopInit</script>class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop

a n o n f u n $ o r g $ a p a c h e $ s p a r k $ r e p l $ S p a r k I L o o p

1 <script type="math/tex" id="MathJax-Element-849">1</script>

anonfun<script type="math/tex" id="MathJax-Element-850">anonfun</script>apply

mcZ <script type="math/tex" id="MathJax-Element-851">mcZ</script>sp

5.apply <script type="math/tex" id="MathJax-Element-852">5.apply</script>mcV

sp(SparkILoop.scala:974)atorg.apache.spark.repl.SparkILoopInit <script type="math/tex" id="MathJax-Element-853">sp(SparkILoop.scala:974) at org.apache.spark.repl.SparkILoopInit</script>class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit

class.postInitialization(SparkILoopInit.scala:108)atorg.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)atorg.apache.spark.repl.SparkILoop <script type="math/tex" id="MathJax-Element-854">class.postInitialization(SparkILoopInit.scala:108) at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) at org.apache.spark.repl.SparkILoop</script>

anonfun <script type="math/tex" id="MathJax-Element-855">anonfun</script>org

apache <script type="math/tex" id="MathJax-Element-856">apache</script>spark

repl <script type="math/tex" id="MathJax-Element-857">repl</script>SparkILoop

p r o c e s s $ 1. a p p l y $ m c Z $ s p (S p a r k I L o o p . s c a l a : 991) a t o r g . a p a c h e . s p a r k . r e p l . S p a r k I L o o p

org <script type="math/tex" id="MathJax-Element-859">org</script>apache

spark <script type="math/tex" id="MathJax-Element-860">spark</script>repl

SparkILoop <script type="math/tex" id="MathJax-Element-861">SparkILoop</script>

process <script type="math/tex" id="MathJax-Element-862">process</script>1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop

a n o n f u n $ o r g $ a p a c h e $ s p a r k $ r e p l $ S p a r k I L o o p

1.apply(SparkILoop.scala:945)atscala.tools.nsc.util.ScalaClassLoader <script type="math/tex" id="MathJax-Element-864">1.apply(SparkILoop.scala:945) at scala.tools.nsc.util.ScalaClassLoader</script>.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org

apache <script type="math/tex" id="MathJax-Element-865">apache</script>spark

repl <script type="math/tex" id="MathJax-Element-866">repl</script>SparkILoop

p r o c e s s (S p a r k I L o o p . s c a l a : 945) a t o r g . a p a c h e . s p a r k . r e p l . S p a r k I L o o p . p r o c e s s (S p a r k I L o o p . s c a l a : 1059) a t o r g . a p a c h e . s p a r k . r e p l . M a i n $ . m a i n (M a i n . s c a l a : 31) a t o r g . a p a c h e . s p a r k . r e p l . M a i n . m a i n (M a i n . s c a l a) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e (N a t i v e M e t h o d A c c e s s o r I m p l . j a v a : 62) a t s u n . r e f l e c t . D e l e g a t i n g M e t h o d A c c e s s o r I m p l . i n v o k e (D e l e g a t i n g M e t h o d A c c e s s o r I m p l . j a v a : 43) a t j a v a . l a n g . r e f l e c t . M e t h o d . i n v o k e (M e t h o d . j a v a : 498) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t $ . o r g $ a p a c h e $ s p a r k $ d e p l o y $ S p a r k S u b m i t

.doRunMain <script type="math/tex" id="MathJax-Element-868">.doRunMain</script>1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit

.submit(SparkSubmit.scala:206)atorg.apache.spark.deploy.SparkSubmit <script type="math/tex" id="MathJax-Element-869">.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit</script>.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

HarmonyOS开发者社区

讨论HarmonyOS开发技术，专注于API与组件、DevEco Studio、测试、元服务和应用上架分发等。

更多推荐

【开发者故事】这款小众穿戴应用，凭什么让7成用户天天打开？

HarmonyOS开发者社区

鸿蒙系统实况窗功能详解与开发适配指南

实况窗是 HarmonyOS 提供的一种帮助用户聚焦正在进行的任务、方便快速查看和即时处理的通知形态。它支持应用将订单或服务的实时状态信息变化在设备的关键界面展示，并对展示信息的生命周期、用户界面 UI 效果等进行管理。实况窗具有时段性时效性变化性时段性：事件或服务需要持续一段时间，有明确的开始和结束，而非单点的提醒或信息。例如打车、外卖等从事件开始到结束需要经历一段时间，属于实况窗；天气提示、待