More Control Parameters

Let us review some of the properties related to Fair Scheduler.

  • There are properties as part of yarn-site.xml which can be used to overwrite the behavior of Fair Scheduler.
    • yarn.scheduler.fair.user-as-default-queue
    • yarn.scheduler.fair.preemption
    • yarn.scheduler.fair.preemption.cluster-utilization-threshold
  • Also there are several properties that can be defined as part of the allocation file (fair-scheduler.xml). We can review properties from this URL.
    • Queue element – Representing queues. It has the following properties
      • minResources — Setting the minimum resources of a queue
      • maxResources — Setting the maximum resources of a queue
      • maxRunningApps — Setting the maximum number of apps from a queue to run at once
      • weight — Sharing the cluster non-proportional with other queues. Default to 1
      • scheduling policy — Values are fifofairdrf or any class that extends org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.scheduling policy
      • aclSubmitApps — Listing the users who can submit apps to the queue
      • minSharePreemptionTimeout — Specifying the number of secs
    • User elements – Representing user behaviors. It can contain a single property to set a maximum number of apps for a particular user.
    • userMaxAppsDefault element – Setting the default running app limit for users if the limit is not otherwise specified.
    • fairSharePreemptionTimeout element – Setting the number of seconds a queue is under its fair share before it tries to preempt containers to take resources from other queues.
    • defaultQueueSchedulingPolicy element – Specifying the default scheduling policy for queues; overridden by the scheduling policy element in each queue if specified.

Share this post