The scent-detection accuracy of dogs (Canis familiaris) was assessed as they learned to discriminate between water samples from aquaria containing koi carp (Cyprinus carpio), goldfish (Carassius auratus) or no fish, at systematically diluted concentrations. The study was conducted using an automated apparatus. Three dogs were trained in a go/no-go task where hits on target (koi carp) samples were reinforced and correct rejections of non-target (goldfish and no-fish) samples had no programmed consequences. Conducting a control test indicated that the dogs’ accuracy measures were likely influenced by extraneous variables. Consequently, modifications were made to procedures and the apparatus. A multiple-probe design experiment found the dogs were able to detect koi carp scent at all of the concentrations tested (from 0.098 mL/100 mL to 0.00094 mL/100 mL), equivalent to an effective biomass range of between 72.5 and 0.4 kg/ha. No clear relationship between accuracy measures and the systematic dilution of the sample concentration was observed for any of the dogs. A test of the dogs’ accuracy with the samples removed from the apparatus suggested that volatiles evaporating from the sampled aquarium water may have been accumulating in the segments as the dogs continued to respond according to the status of the samples that were previously in each segment. Sensitivity was found to be similar across first trials and second trials for target samples. Specificity was found to increase for non-target samples from first trials to second trials. These results have significant practical implications for the use of scent-detection dogs for koi carp detection and the further development of research methods for their training and assessment.