More Control Parameters

Let us review some of the properties related to Fair Scheduler.

• There are properties as part of yarn-site.xml which can be used to overwrite the behavior of Fair Scheduler.
  • yarn.scheduler.fair.user-as-default-queue
  • yarn.scheduler.fair.preemption
  • yarn.scheduler.fair.preemption.cluster-utilization-threshold
• Also there are several properties that can be defined as part of the allocation file (fair-scheduler.xml). We can review properties from this URL.
  • Queue element – Representing queues. It has the following properties
    • minResources — Setting the minimum resources of a queue
    • maxResources — Setting the maximum resources of a queue
    • maxRunningApps — Setting the maximum number of apps from a queue to run at once
    • weight — Sharing the cluster non-proportional with other queues. Default to 1
    • scheduling policy — Values are fifo, fair, drf or any class that extends org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.scheduling policy
    • aclSubmitApps — Listing the users who can submit apps to the queue
    • minSharePreemptionTimeout — Specifying the number of secs
  • User elements – Representing user behaviors. It can contain a single property to set a maximum number of apps for a particular user.
  • userMaxAppsDefault element – Setting the default running app limit for users if the limit is not otherwise specified.
  • fairSharePreemptionTimeout element – Setting the number of seconds a queue is under its fair share before it tries to preempt containers to take resources from other queues.
  • defaultQueueSchedulingPolicy element – Specifying the default scheduling policy for queues; overridden by the scheduling policy element in each queue if specified.