Iceberg小文件合并

1.用法–Flink

  Iceberg提供了Actions来进行小文件合并，需要手动调用执行

import org.apache.iceberg.flink.actions.Actions;

TableLoader tableLoader = TableLoader.fromHadoopTable("hdfs://nn:8020/warehouse/path");
Table table = tableLoader.loadTable();
RewriteDataFilesActionResult result = Actions.forTable(table)
        .rewriteDataFiles()
        .execute();

  这个任务目前仅仅将小文件进行了合并生成大文件，但旧的文件并没有删除，也就是文件反而变多了
  功能一路调用，到RowDataRewriter类中的rewriteDataForTasks

public List<DataFile> rewriteDataForTasks(
    DataStream<CombinedScanTask> dataStream, int parallelism) throws Exception {
   
   
  RewriteMap map =
      new RewriteMap(
          schema, nameMapping, io, caseSensitive, encryptionManager, taskWriterFactory);
  DataStream<List<DataFile>> ds = dataStream.map(map).setParallelism(parallelism);
  return Lists.newArrayList(ds.executeAndCollect("Rewrite table :" + tableName)).stream()
      .flatMap(Collection::stream)
      .collect(Collectors.toList());
}

  这里构建Flink任务，核心是RewriteMap的算子，其map方法如下，本质是读取并写入数据

public List<DataFile