Spark Shuffle FetchFailedException解决方案

原创

已于 2022-11-03 18:22:50 修改 · 4.5k 阅读

标签

#maven #java #mysql

于 2022-03-08 16:39:22 首次发布

在处理一个数据倾斜的SQL时，增大`spark.sql.shuffle.partitions`从500到2700反而引发FetchFailedException。问题在于Spark的Shuffle机制。默认情况下，当Block大小超过200MB时，Spark会流式读取到磁盘，但小于该阈值则全部加载到内存。调整`spark.maxRemoteBlockSizeFetchToMem`为10MB无效，因为当shuffle partitions超过2000时，Spark使用HighlyCompressedMapStatus，仅记录平均值和部分大块。解决办法包括减少shuffle数据、调整分区数、提高executor内存、处理数据倾斜和检查空值过滤。

某日遇到一个数据倾斜的SQL, 首先想到的方法就是加大Partition 看看数据hash 之后会不会落得均匀,所以就将spark.sql.shuffle.partitions从原来的500 加大到2700 .
结果反而失败了, 错误如下:

FetchFailed(BlockManagerId(516, nfjd-hadoop02-node352.jpushoa.com, 7337, None), shuffleId=3, mapId=59, reduceId=917, message=
org.apache.spark.shuffle.FetchFailedException: failed to allocate 16777216 byte(s) of direct memory (used: 2147483648, max: 2147483648)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:554)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:485)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:64)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage8.sort_addToSorter_0$(Unknown Source)
	at org.apache.spark.sql

最低0.47元/天解锁文章