AI Testing of a Trading platform: test data generation, report interpretation and model reinforcement
At this year’s AI in Capital Markets Summit hosted by A-Team in London, I had the opportunity to present our case study on using AI in capital markets to improve functional correctness and resilience of financial market infrastructures.
"Never invest in a business you cannot understand," – the quote attributed to Warren Buffett
The healthy way to own an asset is to know what is inside. If this is the case, you will be able to put the asset to work and get returns on your investment.
At Exactpro we help fintech companies, including a dozen of top global exchange groups, to better understand one of their main assets. Exactpro has clients in more than 20 countries and operates from several delivery centres across the globe.
Technology is one of your main assets
Technology or software is one of the main assets for many modern companies. Both AI and conventional one. Software testing is exploring technology with the intention to find defects - anything that can decrease the value of the software to its stakeholders. Building and operating software without thorough testing is like devising a military strategy without proper intelligence, treating patients without accurate diagnostics or participating in electronic trading without access to market data.
How can one understand something? How do you question the nature of your reality? Here is a diagram of CERN’s Large Hadron Collider. We start with a theory of how things work. We run massive experiments, collect the data and evaluate if the observations reveal something unexpected.
Exploring technology to obtain information
Exploring software is a similar process. We need to decide what we want to do. We need to perform the steps, collect the data and evaluate the outcome. Most people would agree that the inception of ideas and interpreting the outcome is a cognitive effort that requires a certain level of intelligence.
With the growth of available digital data and computational capabilities, we are seeing the use of subsymbolic AI delivering improvements in autonomy and efficiency across many industries. It is possible to use machine learning to enhance the generation of test ideas and the interpretation of test results.
The process consists of the following six steps:
Requirements analysis
Input dataset preparation
Test execution
Output dataset interpretation
Model refinement
Reinforcement loop
Case Study: Trading Platform
This diagram illustrates the practical use of the approach to explore software for an exchange platform. We recently used it for digital exchanges, small MTFs and also for some of the most liquid and visible markets in the world.
The easy part is to connect across multiple order-entry and market data channels, subscribe to regulatory and surveillance feeds. It is relatively easy to initiate a diverse flow of events into the market.
Next, we use property checking and a digital twin for a more difficult task: to evaluate the outbound data. The focus of this particular case study: can AI generate something interesting, original and creative?
Stephen Wolfram, one of the pioneers in exploring the computational nature of reality, answers this question in his recent Forbes interview. It’s actually very trivial to do something original and creative. You just pick a bunch of random numbers.
How do we find the original stuff that is interesting? We need to have a discriminative model to process the randomly generated numbers. In our work we base it on measuring the diversity of the outbound feature sets for generated scenarios.
What is a proper loss function for a generative model? How to evaluate the output?
The purpose of software testing is to provide information – detect and describe defects in the software. The value of the output can be measured across the following three main dimensions:
Speed – testing faster
Cost – testing cheaper
Quality – testing better (*)
When we say “better”, what do we mean? “Better” at what? At detecting and describing defects. In fact, it is easy for AI to detect something. Explaining the generated data to humans is a substantially harder task, and we need it for many reasons. Regulatory compliance is not the least important of them.
Talking about the loss function, there is always a way to optimise both time and money when you do not want the information about defects.
If you want to really understand your software, there are ways to reduce the footprint and complexity of the setup. The digital twin reflects the key aspects of the explored technology. To carry out the process efficiently, one has to consider using a set of the models with reduced complexity. A diverse set of half measures often delivers a lot of pragmatic value.
If you are working on any complex transformation program, if you want to learn how to test AI systems or how to use AI in software testing - please do talk to us.