Stress testing algorithms

Mathematics has the power to influence broadly because it underpins so much. I didn’t understand this when I first decided to become a mathematician. A teacher in Year 12 inspired me to see the beauty of mathematics and I was determined to learn more, without knowing where it might lead.

I decided on my first day at university that I wanted to be a lecturer. I thought it looked like a pretty good job to explain beautiful mathematical ideas to a large crowd.

It wasn’t until I was in my third year at university that I began to see some real mathematical applications. We were learning about partial differential equations and the lecturer said, by way of a throwaway line “... and that equation is used for modelling traffic flow on a freeway”. And I thought, ‘Wow!’ I finally saw exciting opportunities to use my love of mathematics to impact the world.

My current research is focused on a project that began a decade ago. As editor of a journal, I was handling submissions from authors around the world, all wanting to publish their new algorithms that had been tested on only a small set of hand-picked examples. And I thought, ‘That’s not rigorous enough! We need to do better because the implications of untrustworthy algorithms are far too important.’

No one likes to talk about weaknesses in research, but objectivity is critical to trust. Some researchers were showing only where their algorithms succeeded, even though I’m sure it would have been possible to find examples of where they hadn’t performed as well. My interest in ensuring that conclusions were valid and insightful – and not tainted by the potentially biased illustrative examples that researchers had selected to demonstrate their strengths – began to grow.

I saw the opportunity for a paradigm shift in how research is reported. A new methodology that would reveal both the weaknesses and strengths of an algorithm across the broadest “test instance space” became an imperative for me. We’ve developed the mathematical idea to create two dimensional (2D) instance spaces for many problems that can be visualised easily, showing where an algorithm is expected to perform well and poorly, and providing objective statistical analysis of its strengths and weaknesses.

This means we’re ‘stress testing’ algorithms. We have developed a methodology to generate new test instances with controllable characteristics facilitating the creation of the most comprehensive collections of test problems.

I aimed to develop a suite of powerful tools to improve our trust in algorithms through more rigorous testing and my Laureate Fellowship project has been funded by the Australian Research Council for the last five years.

The resulting research platform is almost ready to share with the world. Our website, MATILDA – Melbourne Algorithm Test Instance Library with Data Analytics – is due to launch early next year. It’s an online repository where researchers can download our test instances, test their algorithms, and upload their performance results.

By clicking a button, they can generate a 2D map of the instance space showing their algorithm’s strengths and weaknesses. It will also provide some objective statistics reporting how competitive their algorithm is compared to others. They can then include this in a scientific paper as evidence of their new algorithm’s performance, as well as the insightful analysis our tools will provide. It’s the kind of honest and objective reporting we need in many fields of algorithmic science.

Our case studies cut across different disciplines like optimisation, machine learning, and forecasting – but also fields like Artificial Intelligence where the need for trustworthy algorithms is a hot topic. Seeing the impact of my work to improve research across many fields has been a rewarding spin-off that I hadn’t anticipated 10 years ago when I began this journey to improve algorithm testing.

These days, I enjoy working with industry on collaborative research and consulting, but I’d never trade academia for industry. I love the balance of being an academic, the intellectual freedom to pursue what I think is interesting and I feel I can impact the world from within the university.

I love teaching as well, and training the next generation. It’s a great life and I value what I have. This is the message I share when mentoring students and junior academics, especially women. I advise them to follow their passions first. I stress the importance of keeping everything in balance while striving for the satisfaction that comes from creating a positive impact on the world.

Female role models are critical and that’s why mentoring is so important to me. I never saw a female lecturer in my four years of doing a maths degree. I only learnt in recent years that one of the pioneers in my field, Alison Harcourt, was working in the department while I was a student learning about the algorithm she developed in 1960.

She’s now 88-years-old and still tutoring in my department. She never received the recognition she deserved for her ground-breaking work.

It’s been a personal mission of mine this year to get her the attention she deserves. In addition to being named Victorian Senior Australian of the Year, she’s soon to be awarded an honorary doctorate from the University of Melbourne.

It’s a great start.

- As told to Muriel Reddy

Banner Image: Shutterstock