For submitting non-interactive Spark and Hadoop jobs, we are going to access Owens cluster through the shell app after logging into OSConDemand. Please click on Clusters, then on Owens Shell Acess.
This will open a shell on the Owens cluster where you can enter UNIX commands. First, we are going to copy input files required to your home directory. Please enter the following command,
cp -rf /users/PZS0645/support/workshop/Bigdata/Non-interactive $HOME/
This will copy a directory called Non-interactive to your $HOME. First, we will submit two Spark jobs.
cd $HOME/Non-interactive/spark
Check the contents of the directory using ls command
ls
Please use more command to see the content of a file
more stati.pbs
You can change job parameters like number of nodes, walltime limits etc in .pbs file.
Submit jobs using the qsub command
qsub stati.pbs qsub sql.pbs
Check job status using the qstat command. You will see status under S tab as Q for in queue or R for running or C for completed.
qstat -u `whoami`
When the jobs are completed, it will generate stati.log and sql.log files as output. You can check out the output with more command.
more sql.log
Next, we will submit hadoop jobs.
cd $HOME/Non-interactive/hadoop ls
Submit jobs using qsub command
qsub sub-grep.pbs qsub sub-wordcount.pbs
When jobs are completed, two directories grep-out and wordcount-out will be created. To check results,
cd grep-out more part-r-00000 cd ../wordcount-out more part-r-00000
