Reconstructing Data Provenance from Log Files

Tan, Yu Shyang

Reconstructing Data Provenance from Log Files

dc.contributor.advisor	Ko, Ryan K.L.
dc.contributor.advisor	Holmes, Geoffrey
dc.contributor.advisor	Rogers, Bill
dc.contributor.author	Tan, Yu Shyang
dc.date.accessioned	2017-10-09T00:17:50Z
dc.date.available	2017-10-09T00:17:50Z
dc.date.issued	2017
dc.date.updated	2017-09-22T03:25:35Z
dc.description.abstract	Data provenance describes the derivation history of data, capturing details such as the entities involved and the relationships between entities. Knowledge of data provenance can be used to address issues, such as data quality assurance, data audit and system security. However, current computer systems are usually not equipped with means to acquire data provenance. Modifying underlying systems or introducing new monitoring software for provenance logging may be too invasive for production systems. As a result, data provenance may not always be available. This thesis investigates the completeness and correctness of data provenance reconstructed from log files with respect to the actual derivation history. To accomplish this, we designed and tested a solution that first extracts and models information from log files into provenance relations then reconstructs the data provenance from those relations. The reconstructed output is then evaluated against the ground truth provenance. The thesis also details the methodology used for constructing a dataset for provenance reconstruction research. Experimental results revealed data provenance that completely captures the ground truth can be reconstructed from system-layer log files. However, the outputs are susceptible to errors generated during event logging and errors induced by program dependencies. Results also show that usage of log files of different granularities collected from the system can help resolve logging errors described. Experiments with removing suspected program dependencies using approaches such as blacklisting and clustering have shown that the number of errors can be reduced by a factor of one hundred. Conclusions drawn from this research contribute towards the work on using reconstruction as an alternative approach for acquiring data provenance from computer systems.
dc.format.mimetype	application/pdf
dc.identifier.citation	Tan, Y. S. (2017). Reconstructing Data Provenance from Log Files (Thesis, Doctor of Philosophy (PhD)). The University of Waikato, Hamilton, New Zealand. Retrieved from https://hdl.handle.net/10289/11388	en
dc.identifier.uri	https://hdl.handle.net/10289/11388
dc.language.iso	en
dc.publisher	The University of Waikato
dc.rights	All items in Research Commons are provided for private study and research purposes and are protected by copyright with all rights reserved unless otherwise indicated.
dc.subject	Data Provenance
dc.subject	Reconstruction
dc.subject	Log Analysis
dc.title	Reconstructing Data Provenance from Log Files
dc.type	Thesis
pubs.place-of-publication	Hamilton, New Zealand	en_NZ
thesis.degree.grantor	The University of Waikato
thesis.degree.level	Doctoral
thesis.degree.name	Doctor of Philosophy (PhD)