At this moment I do not have a personal relationship with a computer.
– Janet Reno
The number of Spark executors you need for a variable input file size depends on several factors, such as the available resources in your cluster, the amount of memory required by your Spark application, the number of cores in your cluster, and the size of your input data.
One common approach to determining the number of executors to use is to consider the total number of CPU cores and the amount of memory available in your cluster, and then divide them by the number of executors you want to use. For example, if your cluster has 100 cores and 1 TB of memory, and you want to use 10 executors, you can allocate 10 cores and 100 GB of memory per executor.
However, this approach may not be sufficient if the input file size varies significantly from day to day. In that case, you may want to consider dynamic allocation, which allows Spark to adjust the number of executors based on the workload and available resources.
Dynamic allocation can be enabled by setting the following configuration properties:
spark.dynamicAllocation.enabled=true
spark.shuffle.service.enabled=true
With dynamic allocation, Spark will start with a minimum number of executors and scale up or down based on the workload. You can set the minimum and maximum number of executors using the following configuration properties:
spark.dynamicAllocation.minExecutors=<minimum number of executors>
spark.dynamicAllocation.maxExecutors=<maximum number of executors>
By setting the minimum and maximum number of executors appropriately, you can ensure that your Spark application can handle input files of varying sizes without wasting resources or slowing down the processing time.