checkpoint: truncate a lineage that's gotten out of hand
In iterative pipelines, the logical plan grows until it triggers StackOverflow errors or explosive recomputation; checkpoint cuts the lineage.
Prerequisites
PySpark 3.x
Python
spark.sparkContext.setCheckpointDir("s3a://lake/tmp/checkpoints")
df_iter = seed
for i, rules in enumerate(rule_batches):
df_iter = apply_rules(df_iter, rules)
if i % 10 == 9:
# Écrit sur stockage fiable et REPART d'un plan vide
df_iter = df_iter.checkpoint(eager=True)
# localCheckpoint() : plus rapide (disque executor) mais perdu si un
# executor meurt — acceptable en exploratoire, pas en production.
# Alternative robuste et explicite : write.parquet(tmp) puis read.parquet(tmp).Result
iteration 04 plan: 1248 noeuds batch: 96 s iteration 09 plan: 2730 noeuds batch: 311 s -- checkpoint(eager=True) : ecriture 38 s, plan tronque -- iteration 10 plan: 12 noeuds batch: 41 s iteration 19 plan: 1304 noeuds batch: 102 s -- checkpoint(eager=True) : ecriture 39 s, plan tronque -- Sans checkpoint : StackOverflowError a l'iteration 23.
PySparkcheckpointLineageItératif