The research method used is a systematic review , with demands placed on research questions, identification of research, selection process, appraisal, synthesis, and inferences. We now address each of these in turn.
1 Planning the review
We started by developing a protocol for the systematic review, specifying in advance the process and methods that we would apply. The protocol specified the research questions, the search strategy, criteria for inclusion and exclusion, and method of synthesis.
The aim of the study was to provide an overview of the empirically studied methods for knowledge management in software engineering. To achieve this, we decided to address the following research questions:
(1) What concepts have been investigated empirically within the field of knowledge management in software engineering.?
(2) What are the research methods used in studying knowledge management in software engineering?
2 Identification of research
A comprehensive, unbiased search is a fundamental factor that distinguishes a systematic review from a traditional review of the literature. Our systematic search started with the identification of keywords and search terms.
Software engineering keywords Knowledge management keywords
• software engineering
• software process
• learning software organization • knowledge management
• tacit knowledge
• explicit knowledge
• knowledge creation
• knowledge acquisition
• knowledge sharing
• knowledge retention
• knowledge valuation
• knowledge use
• knowledge application
• knowledge discovery
• knowledge integration
• knowledge Theory
• organization knowledge
• knowledge engineering
• experience transfer
• technology transfer
All possible permutations of the software engineering and knowledge management concepts were tried in the search conducted.
The following electronic bases were searched, using the strategy outlined. The electronic bases were those we considered most relevant : ISI Web of Science, Compendex, IEEE Xplore and the ACM Digital Library.
In addition, we identified two arenas that, to our knowledge, are the only ones that pertain specifically to knowledge management in software engineering: the workshop series on Learning Software Organisations (LSO) from 1999 until 2006, and the book Managing Software Engineering Knowledge . We searched all proceedings from the workshop series and included all chapters from the book.
We performed the search in August 2006, which means that publications up to and including the first quarter of 2006 are included, but some studies in the second quarter might not have been indexed in the databases.
The identification process yielded 2102 articles. This formed the basis for the next step in our selection process.
3 Selection of primary studies
The first step after the articles had been identified was to eliminate duplicate titles, and titles clearly not related to the review. One researcher (FOB) read through the 2102 titles and removed duplicates and those clearly not related to the field of software engineering. This yielded a result of 762 articles.
After this we obtained the abstract of these articles and both authors read through all abstracts, with the following exclusion criterion.
• Exclude if the focus of the paper is clearly not on software engineering
• Exclude if the focus of the paper is clearly not on knowledge management
• Exclude if the method, tool or theory described is not tested in industry
To narrow the search further we also decided to focus on technical and process knowledge (thus, “software engineering knowledge”). Hence, we also used the criterion
• Exclude if the focus of the paper is on domain knowledge
After each researcher had gone through the papers we compared results. Where we disagreed as to whether to keep or remove a paper, we discussed the matter until we reached agreement.
This process reduced the number of articles to 133, and agreement between researchers was ‘good’ (Kappa value of 0,655).
The full text for all 133 papers was obtained and both researchers read through all the papers with the same criteria for exclusion in mind. The final number of papers selected for the review was 68. The agreement between researchers at this stage was “moderate” (Kappa value: 0,523).
4 Quality assessment and classification
We chose to classify the 68 papers identified along two axes. (1) We wanted to examine what kinds of concept had been tested. To aid us with this we chose the framework for classifying strategies for managing knowledge presented by Earl in . Each researcher classified the 68 papers individually according to the framework, before comparing the results. Disagreements were discussed until a consensus was reached on the classification. (2) We also wanted to examine the scientific rigour of the studies. Here we settled on a simpler classification. All studies included so far had results taken from industry. We further assessed the quality of the selected papers by categorising these into empirical studies and lessons learned reports. The criterion for being accepted as an empirical study and not a report of lessons learned was that the article had a section describing the research method and context. Again, each study was classified individually by the two researchers before comparing the results and discussing problem cases in order to reach agreement. After the quality assessment, we had 29 empirical studies and 39 reports of lessons learned.
For the synthesis, we used the papers classified as empirical studies in our framework. We extracted concepts covered and the research method for each article. One researcher (FOB) focused on the studies in the technocratic schools, while the other researcher (TD) focused on the behavioural schools.