Audits, Assessments and their role in the test ing improvement process.

I am a QA Team Leader with seven years of experience, five years of which involved QA team management. I have an ISTQB Certified Tester Full Advanced Level certificate, and I have experience in managing QA-teams (manual, automation, hybrid) of various sizes, including distributed teams, where participants were in different countries.

During preparation for certification exams, I became interested in the topic of audits and test process improvement and began to apply theoretical knowledge in practice.

This article will be interesting for those who have already thought about process audits on the project, started collecting metrics or are just interested in it – QA Managers, QA Leads, PMs and even executive persons. In this article, we will consider how an audit differs from an assessment, and why and how often to conduct an audit. In addition, I will provide an example of using one of the industry-standard test process improvement models.

I am writing this article because I want to share my experience and theoretical knowledge, and because I spoke on this topic at a webinar for the WWC (Women Who Code) organisation in Ukraine. I think that for the sake of better perception of the material, it is better to read than to hear, and interesting questions from the audience complimented my story.

Audit vs assessment – is there a difference?
I became acquainted with the topic of audits and test process improvement models during preparation for the ISTQB® Advanced Level Test Manager exam. Later, I was asked to jointly work on one of the company‘s projects to evaluate their process and suggest solutions.

Our company needed a well-trained and experienced person who could convince the customer that changes must be made to prevent even greater process problems. It was then that I returned to the books and began to study the topic of audits and models for improving testing processes in more detail.

First, I propose to consider what is audit and evaluation (audit and assessment). On the one hand, this issue is controversial, but on the other – everything is quite simple. The two concepts are closely related but different.

When we talk about assessment, we mean getting up-to-date in-formation about the project and identify its strengths and weaknesses and what can be improved. In addition, we will receive a qualitative assessment, expressed in numbers (for example “Documentation“- 3 out of 5).

The audit is also based on evaluation, but in terms of compliance with certain standards and documents (such as external audits to obtain ISO certificates).

There are three types of audits exist:

  1. A first-party audit is performed within an organisation to measure its strengths and weaknesses against its procedures or methods and/or against external standards adopted by (voluntary) or imposed on (mandatory) the organisation.

    A first-party audit is an internal audit conducted by auditors employed by the organisation being audited, but who have no vested interest in the audit results of the area being audited.
  2. A second-party audit is an external audit performed on a supplier by a customer or by a contracted organisation on behalf of a customer. A contract is in place, and the goods or services are being, or will be, delivered. Second-party audits are subject to the rules of contract law, as they are providing contractual direction from the customer to the supplier. Second-party audits tend to be more formal than first-party audits because audit results could influence the customer’s purchasing decisions.
  3. A third-party audit is performed by an audit organisation independent of the customer-supplier relationship and is free of any conflict of interest. The independence of the audit organisation is a key component of a third-party audit. Third-party audits may result in certification, registration, recognition, an award, license approval, a citation, a fine, or a penalty issued by the third-party organisation or an interested party.

I believe that you do not name it as a process, but it is useful to give. That is why I will continue to operate with both concepts: audit and assessment.

Why a Test process audit is needed and how often to conduct it?

So when should you think about auditing?

• You do not have clear information about software quality or bottlenecks.

• The workload on the project is increasing, but the current quality control process does not allow it to be scaled.

• With the existing quality control process, you cannot switch to a different development methodology, apply a different approach to infrastructure management, or make other changes.

• You need to pass product/process certification, and you want to make sure that the quality control process meets the requirements.

It is important to remember that to achieve a stable positive result, result; the audit of processes must
become continuous. This means that from time to time the whole range of actions must be repeated (plan an audit, conduct it, check the results and implement changes in accordance with the results and the desired effect). To do this, you can use the so-called Deming cycle, also called PDCA Model.

Critical components of the testing process
Now that we have understood assessment, audit and its types, it is time to determine which of the test processes are critical. The concept of „critical test processes“ (Critical Testing Processes) is described in the book of the same name by Rex Black and covers the processes that are subject to evaluation and subsequent changes:

• Testing
• Establishing context
• Quality risk analysis
• Test estimation
• Planning (Test planning)
• Test team development
• Test system development
• Release management (Test release management)
• Test execution
• Creating bug reports (Bug reporting)
• Results reporting
• Change management

You can read more about these processes in Rex Black’s book, but we are currently interested in common models for improving testing processes that can serve as a basis for audits.

The ISTQB® Advanced Test Manager exam book describes four industry-standard test process improvement models. I chose one and used it as a basis for future audits. Perhaps another model is better suited for your projects, so I suggest you briefly read the different approaches.

Test Process Improvement models
TPI Next (Test Process Improvement Next).

In my opinion, the most complex and difficult to understand model. The evaluation offers 16 key processes and several levels of maturity for each. There is also a special matrix to make proper evaluation. This model is more suitable for separated QA teams and focuses on a parallel increase in maturity for all key indicators.

In my experience, it will be impossible to get a transparent, clear, and simple summary in a short time. Of course, if we have more than 1-2 days to perform such an audit, and it is a
project that you work directly with, you can take this model and have good results.

CTP (Critical Testing Processes). This model is more flexible, it does not prescribe anything, unlike other models that expect us to constantly continuously improve in a way with proper consequence of actions. But still, I would like to use something fresher (the booklet Critical Testing Processes is a bit outdated) and clearer for people who are likely to hear for the first time about the existence of similar models in general. STEP (Systematic Test and Evaluation Process). All the changes that this model proposes to implement can take place in any order. It has no complex matrices. But one of the
principles is the existence of clear, formalised requirements. That is, for a process where there are none, this model is probably not suitable.

TMMi (Test Maturity Model Integration). The model was developed by the Illinois Institute of Technology based on CMM (Capability Maturity Model). Its purpose is to provide a basis for assessing the maturity of testing processes. And thus increase the level of maturity. TMMi has five levels; they are shown in the diagram:

Thus, typical process areas are divided into groups, each of which has goals (Specific Goals) and means to achieve them (Specific Practices). So, to use this model, we need to assess the means to achieve the goals within a particular project. Of course, this model has its drawbacks. The main one is to perhaps one of the challenges of this model is the need to create a lot of documents to move to the next level, if they are not already within the project.

An example of practical use of the industry model to improve the TMMi test process I will not describe the chosen model in detail, because the official site offers links to documentation and there are already some articles on that in the global network. I just want to share my own example of using the model during a process audit on a project where I did not work directly, but was invited as an independent expert.

This is useful because such attempts do not take much time, give an independent assessment from colleagues, because often we do not notice the little things that can affect the work, product quality and customer satisfaction.

Useful links

  1. Rex Black‘s book Critical Testing Process.
  2. ISTQB Advanced level Test Manager preparation book, where there is a lot of information about process improvement models, as well as comparative analysis.
  3. Official TMMi website, where you can find links to the documentation.
  4. Example of a survey based on TMMi level 2 to 5.

So how did I apply the theoretical knowledge about this model:

• First, I had to assess the level of the project according to the levels of the model.

• After receiving a reasonable result, provide recommendations for the transition to the next level.

Since I was not working on the project, I had to create a list of questions (they may be different, because the model does not require a specific list) and conduct interviews with colleagues who were involved in it to do a survey.

The questions mostly concerned processes, documentation, and so on. Then I added unanswered questions (because this could provoke a biased attitude towards my colleagues on the part of the cli-
ent if they suddenly disagreed with their answers) to the final document, which was then presented to the client.

Then I suggested some steps to take the project to the next level. Formed a separate document for presentation to the customer, which contained the following information:
• A brief overview of the model, levels, and links to official documentation for this model.

• A description of the audit of who was involved in the survey, as well as a list of questions to be answered.

• Conclusion on the current level of the project and recommendations for the next transition. Recorded the benefits for the project and the customer after applying the necessary changes.

Author: Ramella Basenko

A Lead QA Engineer at AgileEngine, has 8 years of experience in the QA area. In her daily work mainly focuses on process improvements and project transformations as well as team management and career growth of QA professionals within the company.

ISTQB Full Advanced level certificate holder. Has degrees in German philology, Business, and Administration. Speaker at conferences and webinars in the field of software quality and
certification of QA specialists.