Currently, the evaluation of network anomaly detection methods is often not repeatable. It is difficult to ascertain if different implementations of the same methods have the same performance or the relative performance of different methods. This is in part due to a lack of open implementations, the absence of recent datasets and no common format to express results. A common approach to evaluating a method is to use the Defense Advanced Research Projects Agency (DARPA) 1999 datasets, or a derivative of them, in combination with a different dataset or network capture. The DARPA datasets are relatively old and bear little resemblance to modern day traffic and the other datasets are unlabelled and typically publicly unavailable making it difficult to ascertain the validity of the research evaluated in such a way. This thesis primarily contributes a new evaluation methodology that uses a data fusion based approach that allows for reproducible evaluations with modern datasets. The new methodology incorporates three other contributions: A new way to capture network traces that are fully anonymised yet retains more information than any current network traces and a new trace annotation format and a method for verifying the correctness of the annotations. The DARPA 1999 dataset was used to demonstrate the validity of the approach and an evaluation was performed on a new dataset that has been captured using the methods introduced. In the evaluation we find that methodology is a viable approach forward, but that it comes with a different set of drawbacks than the current state of the art.