Core Insights: Future of Work

From chatbots and virtual assistants to creating stock portfolios and identifying diseases among thousands of X-rays, Artificial Intelligence (AI) is becoming essential to many organisations.

Indeed, if a company is not looking to adopt AI then they will soon be left behind. And yet there are many companies still yet to realise the potential benefits of incorporating AI into their business operations.

A 2019 survey by accounting giants PwC found that 85 per cent of CEOs felt that AI had the potential to significantly change their business in the next five years.

However, off-the shelf options for plugging AI into a company’s systems are rare, and often the practicalities are so complex it is better off designing a bespoke system.

AI is not just about automating processes and speeding up systems that involve repeated analysing of data. Indeed, with the development of machine learning, AI is now being applied to knowledge-based industries such as the legal profession.

Instead of following rules coded from experts' instructions, machine learning uses masses of data to spot patterns and develop its own algorithmic rules to solve problems and make decisions without human involvement.

Machine learning is better for complex tasks that require human judgement, but the decisions it makes are unfathomable with ‘black box’ reasoning that can’t be explained.

This can bring up issues around transparency for companies, while machine learning also creates problems around defining the actual problem it needs to solve; finding enough quality data for it to train with; and measuring the accuracy of its decisions because the reasons behind them are so opaque.

We spent nearly two years researching how a new legal-tech firm solved these problems in developing a machine learning system that could more efficiently analyse legal documents and historic cases to extract valuable information to use in court. It would save junior lawyers spending hours looking through thousands of documents and books manually.

From our study, here are three recommendations for those wanting to use machine learning in their organisation.

1 Co-formulate the problems

Successfully developing a machine learning system starts with properly defining the problems for it to solve, which is a complex task, but critical and needs senior staff working with AI developers together.

Experts from the organisation need to work with developers on defining these problems, rather than leaving the developers to do it. The company needs to explain the overall vision and then break that down into questions that need solving before the AI developers can then turn that into code. These questions need to have ‘yes’ or ‘no’ answers.

It also means whoever is working with the AI developers needs to gain an intuitive understanding of key concepts. This can be achieved by the developers holding a workshop to explain AI to staff at the organisation and going through the key concepts so they are understood.

For complex problems it helps to break it down into smaller problems to find more simple solutions. For example, to simplify matters the legal tech firm we followed has its AI system ask the same question as it scans each paragraph of a court document.

Despite breaking down problems, in some cases the AI is not capable of going any further and it will need human help, so an AI-human hybrid system will need developing. Or even a new algorithm is needed. In our case study lawyers wanted the AI system to scan whole documents, which can be 6,000 words or more, but state-of-the-art algorithms had a word limit of 512. It took two months to develop a new algorithm that could read whole documents.

2 Refine through iteration

The machine learning AI is always learning, so once it is up and running don’t think it is over, and don’t think it has to be running perfectly before you can use it. Just get it to ‘good enough’ to start using the system, even if some of its functionality is performing poorly, and then continuously refine it.

This is done by supplying it with more and more data, the more data and the more varied good quality data it is fed, the more accurate its decisions will be. For instance, deep learning models often contain hundreds of millions of parameters, so the amount and quality of training data is critical for it to be able to optimise these parameters.

This is why its initial training dataset is so important. Don’t just select a random pile of data, make sure there is an even spread of typical cases and infrequent cases, ie a balanced number of examples of ‘yes’ and ‘no’ for each question. This should iron out any biases and make it more accurate.

However, this is a lot of manual work to get all the data tabulated consistently, so to save time get the machine learning to just ‘good enough’, around 80 per cent accuracy. This can then be reviewed by experts in the field, refined and fed back into the system. For our lawyers this meant using ‘just’ 200 documents to begin with, but to gain more accuracy the training dataset eventually reached 701.

So once it is up and running any biases or imperfections can first be corrected by humans and then by adding data to correct the pattern it is finding.

3 Ask for clues in measuring accuracy

It is very hard to explain the behaviour of machine learning AI, particularly deep neural networks, but the best way to understand its decisions is looking for clues rather than focusing on one metric.

Fixating on one metric can downplay the importance of other numbers. Our lawyers were told the machine learning system had reached 90 per cent accuracy, but they found it had a false-positive rate of 38 per cent and a false-negative of five per cent because of an imbalanced data set.

For our lawyers they could not work with five per cent false negatives and wanted that at zero. The developers adjusted the model to reach one per cent, though that meant a 56 per cent rate for false-positives, which the lawyers could live with.

Thus, it is first important for company executives to discuss with AI developers what metrics fit with their objectives and what criteria they will use to evaluate them. Once they have been chosen it is vital that they are assessed properly, and this is done by using them as clues to a wider trend.

For example, listing the most influential features can provide an insight into how decisions are made by the AI system, while heatmaps that highlight the neuron activations in the text or image, is another method that can offer clues as to the reasoning of the machine learning process. For the legal firm, developers added a feature where the system highlighted the most influential words and phrases that the deep learning model weighted heavily to predict the ‘yes’ or ‘no’ outcome.

Measuring its accuracy is also done by comparing the machine’s conclusions with expected results in a test, so it is vitally important that the results being used to compare it are 100 per cent robust and accurate in the first place. This can be difficult in a knowledge-intensive industry like law where a degree of subjectivity is needed in judging documents and simple ‘yes’ or ‘no’ answers are difficult to come to.

The AI’s results should also be cross-validated with human calculations. If they differ this may not mean that the AI is wrong, but can lead to challenging assumptions made by the firm, so analyse the differences carefully and with an open mind. Our legal tech firm discovered that on closer inspection a number of what were thought to be false-positives were actually true positives that lawyers had missed in their manual review.

Using machine learning for knowledge-intensive industries is complex and needs AI developers working closely with experts in the industry. But by following our recommendations it can help companies speed up their processes and find savings as humans can move onto other less laborious tasks. The legal tech company saw time spent on each file reduced by at least 90 per cent, with just three minutes of analysis by the AI system needed. However, there was also 15 minutes of human validation needed afterwards.

AI is set to revolutionise every industry and for companies who want to be at the forefront of this they need to adopt it with their eyes wide open. For AI to be an advantage it has to be tailor-made to the context of the organisation, which means a heavy investment in time as well as resources.

It will not takeover but augment human expertise and over the next 20 years AI will be the differentiator for those early adopters.

Further reading:

Zhang, Z., Nandhakumar, J., Hummel, J. T. and Waardenburg, L. (2020) "Addressing key challenges of developing machine learning AI systems for knowledge intensive work", MIS Quarterly Executive.

Zhewei Zhang is Assistant Professor of Information Systems & Management and teaches Programming for Data Analytics on the MSc Management of Information Systems & Digital Innovation.

Jochem Hummel is Assistant Professor of Information Systems & Management and lectures on Digital Business Strategy on the MSc Management of Information Systems & Digital Innovation. He also teaches Managing Strategy in the Digital Era on the Undergraduate programme.

For more articles on the Future of Work sign up to Core Insights here.