Spark报错 driver did not authorize commit
来源:互联网 发布:java opencv 二值化 编辑:程序博客网 时间:2024/06/02 11:49
启动Spark Speculative后,有时候运行任务会发现如下提示:
WARN TaskSetManager: Lost task 55.0 in stage 15.0 (TID 20815, spark047216)org.apache.spark.executor.CommitDeniedException: attempt_201604191557_0015_m_000055_0: Not committed because the driver did not authorize commit
启动 Speculative 后,运行较慢的task会在其他executor上同时再启动一个相同的task,如果其中一个task执行完毕,相同的另一个task就会被禁止提交。因此产生了这个WARN。
这个WARN是因为task提交commit被driver拒绝引发,这个错误不会被统计在stage的failure中,这样做的目的是防止你看到一些具有欺骗性的提示。
相关源码
org.apache.spark.executor
case cDE: CommitDeniedException => val reason = cDE.toTaskEndReason execBackend.statusUpdate(taskId, TaskState.FAILED, ser.serialize(reason))
org.apache.spark.executor.CommitDeniedException
private[spark] class CommitDeniedException( msg: String, jobID: Int, splitID: Int, attemptNumber: Int) extends Exception(msg) { def toTaskEndReason: TaskEndReason = TaskCommitDenied(jobID, splitID, attemptNumber)}
org.apache.spark.TaskCommitDenied
case class TaskCommitDenied( jobID: Int, partitionID: Int, attemptNumber: Int) extends TaskFailedReason { override def toErrorString: String = s"TaskCommitDenied (Driver denied task commit)" + s" for job: $jobID, partition: $partitionID, attemptNumber: $attemptNumber" override def countTowardsTaskFailures: Boolean = false}
0 0
- Spark报错 driver did not authorize commit
- CDH5中spark写mysql报“Did not find registered driver with class com.mysql.jdbc.Driver”错误原因
- did not find a matching property (tomcat+Eclipse 报错)
- SVN commit/import报错:405 Method Not Allowed
- Spark运行报错:ERROR CoarseGrainedExecutorBackend: Driver disassociated ! Shutting down
- [iOS 报错] LunchImage 加载报错 "LaunchI did not have any applicable content.
- TOMCAT 关闭报错:Tomcat did not stop in time. PID file was not removed
- 解决Driver not loaded报错的方法
- pdo 连接数据库 报错 could not find driver 解决方法
- 使用了QSqlDatabase数据库,报错:Driver not loaded
- git pull的时候报错:git did not exit cleanly (exit code 1)
- Adapter报错:The content of the adapter has changed but ListView did not receive a
- Adapter报错:The content of the adapter has changed but ListView did not receive a
- Eclipse报错:Setting property 'source' to 'org.eclipse.jst.jee.server:fzxt_tj' did not find
- git:git add readme.txt报错 pathspec 'readme.txt' did not match any files
- 启动tomcat报错:Setting property 'source' to ' ' did not find a matching property
- Cocoapods pod Install报错did not set the base configuration of your project
- git:git add readme.txt报错 pathspec 'readme.txt' did not match any files
- iOS 点滴积累
- Remove Duplicates from Sorted List
- 【fedora】设置中文为默认语言
- 【干货】Spring MVC与JAX-RS比较与分析
- ios点击空白界面退出键盘问题。
- Spark报错 driver did not authorize commit
- JavaMail技术 用java代码发送邮件
- IAP内购整个流程
- 【leetcode】125. Valid Palindrome
- python2 与 python3 语法区别
- easyui,datagrid获得总行数,无视分页
- andfix增量升级更新
- JAVA虚拟机之一:垃圾回收(GC)机制
- ETL技术入门之ETL初认识(一)