First, let us run jobs without specifying any queue and see what happens.
Submit the Long running job to production queue by using the below command. This job requires 279 containers.
https://gist.github.com/dgadiraju/f6aa41383703a8b6bb3bb902a8726581
Submit the other job which requires only 18 containers to process the data.
https://gist.github.com/dgadiraju/85c9d016bca952d4b1d5d52a828bff2f
Submitting jobs and Validating Capacity Scheduler
Let us submit few jobs and see how the resources are allocated using Capacity Scheduler.
- Submit the Long running job to the production queue.
- Submit the other jobs to the test and production queues.
- Submit another job to the qa queue
https://gist.github.com/dgadiraju/479059c8e3d7fd1be7d5b7672a76f447
- Now the production queue has two apps. Since it is a Capacity Scheduler the two jobs will be executed once the resources are available in the production queue and the job in the test queue will be executed as it is with the required resources.
- Here is how the jobs will run with Capacity Scheduler
https://gist.github.com/dgadiraju/9da9f833e6e21d6e9fad3cbbdf680dd1