\chapter{Symphony Benchmark S.O.P}\label{BenchmarkingSOP}
Symphony is a computer cluster hosted by The University of Waikato. Having 92 computing nodes, the Symphony cluster
enables researchers to perform a variety of computing tasks. As all the computing resources of the Symphony cluster are
cosidered as a pool, a queue is used to accept the different requests from users.
Each computing job usually defines the programs and required computing resources.
Based on the cluster status, the Symphony scheduler submits the jobs to the queues and then execute them simultaneously.
The following steps are used to benchmark a JStar program on the Symphony cluster.
\begin{enumerate}
  \item Prerequisites
  \begin{enumerate}
    \item Register as an Symphony account. As the Symphony computer nodes are Linux machines, the
    \keyword{SSH (Secure Shell Client)} is necessary to copy our JStar files to/from the Symphony head node.
    \item Install JDK 1.7. Running the JStar program requires Java version 1.7 or above. The installation is described as follows:
    \begin{itemize}
      \item Download the latest JDK for 64-bit linux from Oracle Java website.\footnote{\url{http://www.java.com/en/download/linux\_manual.jsp}}
      \item Unpack the tarball file to the HOME directory.\footnote{\texttt{tar zxvf jdk-7u-version-linux-x64.tar.gz}}
      \item Add \texttt{JAVA\_HOME} variables to the profile file (\texttt{~/.profile}) with the following commands:\\
      \texttt{export PATH="/home/youraccount/jre1.7.0\_09/bin:\$PATH"}
      \item Save change to the profile and run the 'source' command (\texttt{source ~/.profile}).
    \end{itemize}
  \end{enumerate}
  \item Compile the JStar program.\\
  After compiling the JStar program to the Java source code, we archive these generated Java source code into a JAR file
  and upload the required libraries to the \texttt{lib} folder at the same directory of the Symphony head node by
  using the \texttt{scp} command.
  \item Use the same benchmark configuration.\\
  Reliable experimental results are critical to analyze the performance of a JStar program, so each benchmark
  experiment is repeated 30 times with the same JVM options to get steady results.
  \item Submit the benchmark experiments to the Symphony cluster.\\
  A shell script is employed to run all the benchmark experiments. To submit this shell script to the Symphony queue
  we will use the \texttt{qsub} command and specify the required resources. The following example is the configuration
  of the 8-core machine with 8GB of memory.\\
  \texttt{qsub -N PVWATTS -m abe -l walltime=03:00:00,nodes=1:ppn=8,pmem=8g run.sh}
  \item Summarize benchmark results to a CSV file.\\
  Benchmark results are output to the same plain text file. And we use an AWK program to read the file, extract the
  execution time for each benchmark and writes the time out a CSV file, sparated with \texttt{Tab} value.
  \item Plot the speedup graph.\\
  The CSV file is imported to the \texttt{LibreOffice Calc}. As the JVM needs the time to warm up its HotSpot,
  we ignore the results of initial 6 runs and average the execution times in the later runs. Then we plot
  the absolute and relative speedups versus the number of threads in the pool on a chart.
\end{enumerate}
\cleardoublepage