What’s cooking around AI and Software Testing

We have for quite some time heard about Artificial Intelligence (AI) and how it’s about to change everything we know and what we do, not only in our testing life’s but in all aspects. However, in the normal working day there are still very few people I know of are using AI as a tool or with software that has that capability. Yes, for sure if you are working with chat bots at customer services or self-driving vehicles or using Alexa/Siri/Cortana enhanced application etc. but most people I know does not work actively with AI or concepts as deep learning. Yet!

For one year ago QualityMinds we arranged a “test charter” to Dublin in Ireland for the conference Quest for Quality. A test charter is our own unique concept to enjoy a conference even further, where we go with a number of people that will have an all-inclusive experience, including the travel, hotels, meals, special meetings and continuously sharing experience throughout the trip. Due to corona situation we will not do any trip this year but we are looking forward for next year for sure.

Last year main topic was AI and Quality Assurance. As in every conference there were high and lows. One thing that really fetched into my head was the numbers of jobs that will be lost, due to that machines get more powerful and will take decisions themselves. Another thing was that this new technique will even produce more jobs than before, but the entrance level for those jobs will be much higher than the jobs that it will cut. That means in general, a huge shift in world job market is needed and it will benefit those who come out from start with the new competence fresh. According to World Economic Forum the estimated numbers is that about 75 million jobs worldwide will be lost and 133 million will be created.

For testing it has been a long discussion that has focused on that testing will not be needed when software are produced using AI. I would just say that is not how I think it will be and it was also a topic brought up several times at the conference last year. The way of testing will change, but the need to make sure that a software is fit for purpose will still exist, maybe even more important when dubious algorithms has created the code to other programs that are created in the same way. I think maybe the traditional programming roles are more in danger as models and tools will create code instead of typing it in an advanced text editor (aka coding studio).

For testing, a number of new roles is about to be defined and in a few years most likely dominate the market as more and more software will be created using AI.

Here are a few suggested roles and their high level tasks:

  • AI Data manager – A role that will collect, filter, change and adapt the data needed for a specific software purpose.  This role will be critical and most likely there will be good tools (driven by AI) that support the AI Data Manager role to select the “correct” dataset. Data is and will be expensive in many forms, which causes that the role will affect projects budgets etc.
  • AI model trainer – Instead of specifying and actively interact with software. An AI model trainer will need to secure the data, sequence the data to enhance the model (aka software) to do the intended purpose.
  • AI model tester – As with everything. The persons involved in pure development shall always get help with un-biased views. The AI model tester can use the same toolbox and concepts as with non AI based software, but also in this area there will be huge development to support the model tester.
  • Test Manager – In it’s core it will be same as today, that means to make sure that a test strategy is valid and followed, set the targets for which tests that shall be concluded, when and by whom, planning resources, follow up and report etc. But todays tedious tasks that we still do manual or semi-automatic will be made by small bots or tools that collect data, analyse it and report to the persons or bots needing it. The test manager needs to be dirigent of model creations and most likely act as the sponsor of the development. Hence, therefor it might be that this role will have a new name that doesn’t include only test in it.
  • Data tester – A role that will support the AI data manager in verifying the data. As data quality will be one of the major factors to succeed with anything.

Except for those “new” roles that are needed, most of the old ones will still exist, such as security testers, performance testers and other non-functional testers. Acceptance testers need to include much more ethics, ethnics and understand how data bias can affect their testing. Tools and test environment engineers will still be needed but changed competence.

One of the first phases, that will do real change is the topics regarding tools. Yes, there a number of tools available today, but for most people they are solving niche questions. I expect to see a number of different functions that AI will support our daily work in a great way:

  • Test suites. Using data and models from similar projects and techniques, will be provide great support for test suite optimization. Finding ways to priority and remove test cases will easily be a great show case that can open AI for more areas. Money talks!
  • Log analysers. Using machines to find patterns in complex log flows is already existing today and while there might be tools out there doing the work today, I foresee a number of new tools that will support us even better.
  • User behaviour. For an example, as app:s for mobiles are used by billions of users, there is a lot of data that could be collected around behaviour. This data could be used for modelling new apps or to verify usage automatically by using AI tools. As many new users will be bots, the bots behaviours need also to be taken into account. For user behaviours that are human based, we will have much better support, finding patterns and personas.
  • Data selectors. Today in the automotive industry for an example, you can buy datasets from thousands of vehicles that are out driving in every kind of conditions with a lot of mileage. Tomorrow, much more data can be collected and then there will be tools that are helping you to get “good” or “valid” data. Your problem might not be to get data, but that you don’t get the “right” data or how you can select the right dataset. Data selectors that include ethics and moral parameters will have a huge role in the future.
  • Predictive analysers. Code, models, integrations, storage, security, data, antipatterns etc. that finds things before they are released or finding it in a production solution. This is a little bit as the movie “Minority report” where Tom Cruise has a special team that can foresee crime before it happens.  With faster loop times, there must be better support in this area, and where there is a large market, there will be solutions.


In QualityMinds we are all looking forward to get new perspectives and knowledge of the todays and tomorrows questions and solutions. We have taken our first steps into AI as such, some persons more than others due to their consultant assignment. As for all companies out there, with IoT, faster developments cycles and a much more software that needs to be connected and working, we need to continue enhance our competence.

And yes, there are a lot of good resources, even freebees such as https://www.elementsofai.com/ where you can get a real university diploma in AI. But in general, crisp concrete information are still not there, which makes a lot of things still a little bit fuzzy, where you more or less needs to be on an academic researching level instead of having good solutions at your hands. So while the business is getting mature, please let yourself the service to start understanding the concepts and your own learning paths today. Do we even going to see a new syllabus from ISTQB regarding AI or will it be included in all others? Maybe certificates that judge AI from certain protocols will be needed.

For us, we will need continue follow and try to review new concepts or tools as they emerge, as I noted in the beginning. The big change is not really here yet, but for everyday small questions gets their solutions.

Because in the end, with AI doing a lot of  the boring tasks and new interesting services for the mankind will evolve, and in the end, they must work, which means that quality must be kept, that in turn lead us to the subject of software testing and its future.

And the future looks bright for testing. Very bright indeed.

//Bengt Augustsson
Bengt has worked over 20 years in the testing business and is a co-founder of QualityMinds