At the human-computer interface

Chris Hatzis
Eavesdrop on Experts, a podcast about stories of inspiration and insights. It's where expert types obsess, confess and profess. I'm Chris Hatzis. Let's eavesdrop on experts changing the world - one lecture, one experiment, one interview at a time.

It's remarkable how quickly the internet has grown from a mishmash of clunky html pages to a vast network of interactive sites that are so much a part of our daily lives. Modern society simply wouldn't function without an easy-to-navigate web. But most of us don't spare more than a moment's thought as to why, and how, it's become so easy to navigate. Was it our behaviour online that changed web design? Or have its designs been changing us?

Ben Shneiderman is a distinguished professor in the Department of Computer Science, and Founding Director of the Human-Computer Interaction Laboratory at the University of Maryland. He has been ahead of the computer user interface curve for quite some time.

His teams were early explorers of the space between science and social, working on touch screens, hyperlinks and treemaps before these things were even on the periphery of our lives.

He recently visited the University of Melbourne and gave a public seminar titled "The New ABCs of Research: Achieving Breakthrough Collaborations", for the School of Computing and Information Systems, Melbourne School of Engineering.

A pioneer of human-computer interaction and user interface design, Professor Ben Shneiderman explains to our host Steve Grimwade that the user experience manipulates human behaviour as much as we manipulate it, if not more so.

Steve Grimwade
Ben, welcome to Melbourne.

Ben Shneiderman
Thank you, and great to be here.

Steve Grimwade
Now look, yesterday was almost 40 degrees Celsius and if nothing else, Melbournes are suckers for punishment. So as the thermometer goes up, so does the likelihood that you will be at a barbeque. So let's imagine you're at a barbeque, it's stinking hot and someone asks you that everyday question; what do you do? Is there an easy way of describing what you do?

Ben Shneiderman
Yeah, I've been through that one before and I say I'm a computer scientist, but I study the way people use computers and my goal is to make it easier for people to get what they need done. So it's the design of what's become known as the user interface, or the user experience. We're really more into shaping people's experience, getting them past the frustration on to doing what they need to do and then on to the thrill of success and discovery and making something that they want done. Creating a video, making photos, writing poems, making music, getting people to be creative is what it's all about. The goal of good design is to make more people more creative, more of the time.

Steve Grimwade
In your past, you've described the desire to develop pro-social applications for technology; that change the world for good. What are you most proud of?

Ben Shneiderman
Well, I think that's right, that we as technologists have had a profound impact on the world in my lifetime. It's gone from being disconnected to being hyper-connected. It's gone from being alone and struggling, to be social network connected and working together in large teams of people. So I've contributed to things like the touchscreen keyboards, the touchscreen links, the hyperlinks and that enables people to get medical information, be in touch with their families, do business, all kinds of wonderful things that are important for their lives.

Of course, it also brings the cybercriminals, it brings the terrorists, it brings the scammers and the hackers, and there are real problems, and the fake news; let's get to it. There are a lot of problems that are brought by any successful story like this one of the World Wide Web and mobile devices and social media that are so remarkably transformative in the last decade or two decades. It's just very gratifying to see it happening, very troubling to see the downsides.

Steve Grimwade
To that point - I'm probably skipping a fair way ahead now - I mean the irony of social media and of network connections is that social interactions potentially have suffered; that the one-on-one experience has suffered. Would you agree on that and how does technology ameliorate that?

Ben Shneiderman
It's a troubling story, yes. We all see people walking around eyes down onto their screens, fingers and distracted so much that they trip over things, or walk into traffic, so they're deeply involved in their own world. There is a danger of isolation. My dear colleague Sherry Turkle has a book called Reclaiming Conversation that suggests we need to get back to having students learn to talk to each other. So yeah, there are dangers. On the other hand, I think people are more connected with more people. I am in touch with cousins around the world who I just wouldn't be able to do that. I'm in touch with my sister far away on a daily basis. I'm in touch with my daughters who are a continent away and I can be part of their lives and I can see their kids growing up and so there's some wonderful things about it.

I think we're still - everyone's still learning how to put this to work in a balanced way. I think it's good that there's a lot public discussion and we need to think about it. I'd say also, there's real dangers of cyberbullying and lying online and all kinds of problems that occur. So those need to be addressed as well, but overall, the technologies that expand the options for people, I see as generally a good thing.

Steve Grimwade
I think we'll, at a later point we'll skip ahead to the danger of algorithms. But let's go back a little first. Let's talk about you as a younger researcher yourself and I'm interested in the idea of when did you first recognise a gap between science and research and society?

Ben Shneiderman
I was raised in a world where I had lots of different inputs going in; my parents were journalists, my sister was an English professor, my uncle was a legendary photographer and I had all these different parts of my life and while I got into studying computers and mathematics and physics, I was also reading poetry and making photography and all those different things.

Steve Grimwade
You were the black sheep of the family being the computer scientist.

Ben Shneiderman
It was - yeah, that's kind of a healthy form of rebellion of going into these things. These different things. So I chose to make computing my profession and photography became a serious hobby which it remains so. But I was always - maybe more than others - acutely aware of the human side of the technology and I was very much influenced by Marshall McLuhan, who I went to meet and understood his powerful visions about how technology was reshaping the human experience. So I was very devoted early on to that and studied psychology as a student, as well as computing, and in 1980, I wrote a weird book called Software Psychology.

Publishers thought this was kind of strange, but one publisher took a chance on it and put out this book called Software Psychology, 1980, and what happened was pretty amazing. At that time, there were two computer science Book Of The Month clubs and both of them took that as one of their selections. So suddenly, there was an awareness and there was interest in that.

So that was very gratifying to me and we moved pretty quickly onwards. In 1982 a group of us formed a little conference to try and bring together psychologists, computer scientists, others who were interested in these things and we hoped we'd get two or three hundred people and 906 people showed up which was quite amazing. That conference triggered a 35-year history of conferences and the next one will be in Montreal in April of 2018. So in those early days, there were a hundred research papers a year about the human-computer interaction, man-machine interaction it was called at the time, and now there are more than 12,000 papers a year on this; now a topic.

So one of my great satisfactions and then many people give - acknowledge - my important role, is of building that community and I see that now in universities which have degrees in this topic and courses and textbooks and it's a thriving academic area and a thriving business side; the user experience design community.

Steve Grimwade
UX is everywhere now. Sitting next to you, I seem to be so surprised why hasn't this come earlier? Have there been hurdles along the way?

Ben Shneiderman
Yeah, I would say it grew organically from the technology at the time. The early work on design of cockpits and airplanes, even the time of World War II, and in cars, and in other technologies, control rooms were early applications, but it only began to really take off when personal computers arrived in 1980, let's say. It accelerated when the web appeared in 1993. Suddenly, we went from a narrow specialised community of users who were doing it for work, to kids who were playing music and playing games and that was the transition that opened up, and the fact that 8 billion people benefit from the work of this rather modest size community of researchers, is just a great satisfaction.

Steve Grimwade

I saw a video of you, I believe, from maybe the '80s or '90s and there was a bearded young man who would have looked right at home in Silicon Valley, so why weren't you interested in inhabiting the corporate world rather than inhabiting the world you're in at the University of Maryland?

Ben Shneiderman
Yeah, I was pretty happy at the University of Maryland. It was good home base, but you're right. I did like working with companies, I did spend a year sabbatical with IBM. I did consult for AT&T and Apple and General Electric and one after the other and I always had that connection and I think that's shaped me very strongly up to the current day.

So I believe working on real problems that you get from business and government relationships gets you better research, gets you better ideas, fresher thinking, newer inventions. So that's where I think the fun is.

Steve Grimwade
This is potentially the tension at the heart of your work maybe or maybe not? Maybe it's a tension you ignore, but I always thought that one of the greatest tensions at a university was that between pure and applied learning and research. I was going to talk about how shooting for the moon gave us Teflon-coated kitchenware, but I've since realised that actually didn't happen. Sure the NASA space program gave us plenty of spin-offs, but since 1958, American law stipulated that NASA's research and discoveries should benefit the general public and I'm wondering when did applied research begin to push pure research out?

Ben Shneiderman
Bravo, wow. That's a pretty strong statement.

Steve Grimwade
Has it, may be the question?

Ben Shneiderman
Well, that's right. So you've brought me exactly to the battle I'm fighting now of promoting exactly that fresh way of thinking because you're right that going back 70 years, the model of how what academics do is work in their labs, on their own ideas and then somehow that basic research finds applications and goes to market. That was the legend of what it was supposed to do.

But, over the last decades, many people began to notice this isn't really working. I've joined that group because I saw what happened to my own work where working on real problems got me better basic research. We know that that's historically what happened for many important inventions; Louis Pasteur, the famous case of he worked on fermentation of wine that went bad and spoilage of milk and what does he get? He gets the germ theory of disease, he fixes the milk supply, he fixes the wine and he gets vaccinations and big scientific breakthroughs.

So often, they say that necessity is the mother of invention, but necessity is also the mother of discovery and the discovery of basic things, of basic research ideas, often comes by working on a real problem. So my story these days is to convince universities, and students, and researchers, to change their way of thinking to get to work on something real. Get out of the lab, get out of the campus, go talk to people in business, go talk to people in government and non-governmental organisations, find the right kind of partners, find the right kind of problems, work on those and then go for the twin win. Applied and basic combined; ABC. That's the tile of my book; The New ABCs of Research.



Steve Grimwade
Well, we segued into that very nicely without any effort at all. We didn't want to, we didn't even need to, but yet…

Ben Shneiderman
So, I talk about the twin win is the goal. The twin win is you publish papers with theories in respected scientific journals and you get a validated solution that's ready to travel, ready to be implemented. So that's where I think the sweet spot is and I'm trying - working with a group of about 20 universities in North America to try to bring about a shift of attention because I think you're right, your suggestion and your question suggests that the academic style of the past was to work in the lab on your own ideas, be blue sky and be curiosity-driven for as long as you want and that no longer cuts it with me.

I mean, good things may come, but if I were giving advice, as I do for young people, as I'm going to do later today at the University of Melbourne, I'm going to tell them, find yourself partners and people with real problems. I mean, academics are smart people, they just don't always know how to choose the right problems. By working with someone who has a real problem that they care about, whose solution could bring benefits, the researchers start higher up on the pyramid. You get to somewhere stronger sooner.

I've been collecting evidence on this and just in the last few weeks, here is the evidence that I think is really sweet that for the papers that are just done by academics, you get a certain number of citations of people who notice your work, but the papers that are academics with somebody in business and government, you get twice as many citations; more people notice. Citation counts are one of the ways that we measure the importance and the impact of an idea.

Steve Grimwade
Well, there's the word; impact. I guess that's leading the charge of this. I guess I'm not an academic, I'm just a member of the general public. I am aware that people like Einstein gave us pure research and it wasn't applied at all, so I wonder what's the place of pure research still? Is there a place for it?

Ben Shneiderman
Sure, I think there are going to be people who will have ideas that they want to pursue and that may work out and we should listen to some of it, but I think we want to shift or swing the pendulum a little over. Einstein is often pushed back at me as a "what about Einstein?" Well, you remember Einstein was a patent examiner. He was a very practical guy, he did a lot of other work besides these esoteric things, the general and special theories of relativity, but he was also guided by his very practical thinking and his experience from working in as a patent examiner.

So I think there's a real good blend. Having the exposure to the real world problems will make you a better researcher, whether you choose to go down the road of pure research about some phenomena that's not been explored; those may pay off, but I'm saying if you choose something which has a natural path towards making a difference, towards having an impact, you're more likely to get that basic research breakthrough.



Steve Grimwade
Let's move to collaboration and Jeremy Singer wrote of your work that scientists, engineers and designers must transcend academic disciplines to work together, blending their experience and skills to tackle these major 21st-century challenges. I guess my question first is what's at the heart of collaboration? What do you need to do to be a successful collaborator?

Ben Shneiderman
There's a lot of parts to it. Collaborations often fail. But, when they win, they win big. So if you want to win big, you do better if you collaborate. The evidence is really strong on that one; that teams of people do much better than individuals. It's not just more people, it's better work, better thinking because when you're in a team and you have to explain what you want to do to the person you're working with, you've got to be clear about it. You've got to shape your ideas better and that's one thing that helps. The pushback you get is another positive thing.

I have a wonderful 30-year collaboration with Catherine Plaisant. She's been a close collaborator for all these years and our times together are not quiet. We have strong differences, but when she differs, I listen because I know she's got something wise to say and I'd better pay attention. So keeping your mind open to what others say is important.

Now, you touched on an important issue about the collaboration bringing different skills to bear and that's important too. But it's not just going across disciplines that makes you a winner. What makes you a winner is working on a real problem. So we as computer scientists and working with my students, but we've done projects for people in the Department of Biology, in economics and sociology, even English literature, in transportation and working on their problems gave us fresh challenges and we wind up making breakthroughs and publishing papers in the computer science world and we get to work with them and publish papers in their journals. I love doing that and just seeing how the work we do gets to solve other people's problems makes me very happy.



Steve Grimwade
See, you've got to open to the unexpected a lot of the time.

Ben Shneiderman
That's true. I think so. You want to move forward a little bit, take a look around, figure out what's going on and pay attention to the unexpected, yes.



Steve Grimwade
So that non-linear pathway's got to be a difficult one to open yourself up to. You've got to probably bring a mindset to be able to do that. What's the mindset you've brought to it?

Ben Shneiderman
Sure, I think exploration is fun. Getting out on the trail when you're not quite sure what's around the next bend, is a pretty good time. I like doing that. You have to know you may fail, you may come up against a roadblock and have to turn back and backtrack a little while to get further up the hill, so finding those things and understanding that is important. So I'm careful about trying different things and saying I'll give it a day, or a week, and if it doesn't go, onto the next thing, I've learnt something and move on. There are times you get into failures or dead ends, but I think we sort of make a little progress and that takes is to a new clearing in the forest and then we can see the next step forward, the next path up the hill.

Steve Grimwade
Sounds like you invented agile methodology as well, but when we think about the act, or methodology, or design of collaboration, how has it changed over the last few decades? Has it changed?

Ben Shneiderman
Absolutely, I think the evidence is nice and clear about that. Fifty years ago only half the papers published in the literature were collaborations. They were usually single author, that was the general style and now 90 per cent of the papers are teamwork papers. Even there are teamwork papers with more than a thousand authors. There are some papers with a list of authors which is longer than the paper itself. So those are rare beasts, but that is happening.

So the capacity to collaborate has grown dramatically. I think particularly the World Wide Web and Internet and social connectivity, so we find more collaborations, but the collaborations are more effective. The tools of Skype and video communication, of shared documents, of sharing data sets, of sending email to a group of people, all those things have accelerated the processes of research, raised the expectations of what we can do, gets us further along; it's quite dramatic.

Forty years ago, the typical scientist read 175 research papers per year. Now, they read more than 400 a year. The way they get them is very different from 40 years ago, they got six or seven paper journals mailed to their office which they flipped through and that was most of what they knew about and now very few people get paper journals any more. They get notified of advances by email, by listserves, by somebody sending them a note, by a YouTube video, by blogs, by Wikipedia, so the social media have transformed it and there's new strategies.

So what you're saying is, and I think very right, that researchers need to learn these new strategies. They need to learn how to get information, they need to learn how to ask for help, they need to learn how to tell the stories of their successes and partial successes, so I ask my colleagues to spend an hour a month making a video, writing a blog, writing something that gets their stuff out there and ways that they didn't in the past.

Steve Grimwade
So they what, speak out of their silo, I expect as well?

Ben Shneiderman
Indeed, they used to say "my research speaks for itself" and that's no longer true when there are 2 million papers published a year. You'd better be out there and interpreting your work for others, projecting into different communities and that's another advantage of teamwork, is that if you team up with different people, they'll have different media, different networks and the news can spread and then one day someone says, "oh that idea is something I can use" and they're off and running and new things happen.

Steve Grimwade
The rigid researcher is dead; long live adaptability.

In 1987, you created the golden rules of interface design and they've sat at the heart of much of your work and much of many other people's work since then. Given that your work has broadened with all these collaborations, have you had to adapt these principles?

Ben Shneiderman
Yeah, right, it's interesting how things have evolved over time. What you are referring to is in a moment of inspiration, I coined the phrase the Eight Golden Rules of User Interface Design and it actually appeared first in the first edition of my book called Designing the User Interface in 1986 and I put them down there because I was getting a lot of pressure to distil everything into one page and I tried my hand at it and it was okay but each edition, it has changed. So 2016 was the sixth edition of this book which is among the leading books for this field and by now, I have five more co-authors to help me keep this thing going, but it's evolved over time.

Steve Grimwade
Do you think those principles go to design more broadly outside of computer design?

Ben Shneiderman
People have certainly applied them that way, so one of the, I think, interesting ones was about early on it's make it easy to repair errors. Okay, pretty reasonable thing. But as we went over time, it now says, prevent errors. So we found that while making it easy to fix an error was okay, how much better it is to prevent an error. So typical example of that, in the old days, if you had to enter your birth date you had to type whatever your year number and the month and the date, you might inadvertently instead of typing 7th December, you hit 77th December and you get an error message, "illegal date". Well, that all goes away if you just put up a little calendar on screen and people click on that date. They can never enter an incorrect number. So that was part of these larger principles that I coined called direct manipulation as you select from things on the screen rather than typing and where possibly want to shift attention to selection rather than typing.

Steve Grimwade
Design itself is often more craft than science in many ways, but the principle of design puts the human at the centre of the problem and I'm wondering how much the principles of design are valued in a university setting?

Ben Shneiderman
Design itself is still the new kid on the block. Science has been around for a long time. Engineering as a research discipline is a lot newer. In the United States, we can look at the National Academy of Sciences founded in 1863 by Abraham Lincoln and the National Academy of Engineering only comes into existence in 1964, 101 years later, because up to that point engineering was thought to be somebody who runs a train or does something very practical, but engineering after World War II became a research discipline.

Now, part of what I am promoting is the idea that there should be a National Academy of Design by the year 2065, 101 years later. Design, design thinking, has become a strong part of what's necessary to be taught at universities. That's a big shift for some places, but we have a lot of powerful examples of traditional design schools really booming. In the US, Rhode Island School of Design, design schools in many places and the Stanford D.School, for design school, have become prominent places and more and more universities are adding design as part of the curriculum. That's good news because design thinking is a good complement to the scientific method and the engineering method.

Design thinking opens your mind in a divergent way, to challenging the assumptions, to looking for bold ideas, to thinking about the human dimension and these days, the problems we face are much more, shall I say, socio-technical or design-orientated thinking, healthcare delivery, energy sustainability, environmental preservation, cybersecurity, community safety, all these things have a social and a human dimension and a straight scientific method is not going to be enough. So opening up ourselves, and our students, to thinking about design is really an important part of the next decades.


Steve Grimwade
If we look at innovation and maybe it's - well, firstly what's at the heart of the innovation mindset? I guess you've addressed that in a number of ways, but would you look at it now?

Ben Shneiderman
Well, it's hard to be too generic. It's often said, invention is one item, but innovation is putting it to work and to market. So there's a lot of things that take us from a discovery or an invention to an innovation. I think it's understanding a real problem comes back to it, being in touch with the people who can actually intervene and then taking the courage to try something.

So Google has what they call the hybrid method of research. They don't push their brilliant mathematicians and programmers into labs, they put them to work writing code that gets put into real systems, tested on real users, refined over weeks and months and getting out there intervening in the real world is an important part of this message. That's what researchers should do, that's what students should do.

My students during the semester, I expect them to work on a real problem in teams with someone outside the classroom that benefits someone outside the classroom and with the goal of producing a result that survives beyond the semester. I think that the powerful tools our students have means that they can produce societal benefits during a semester, during their education, not when they graduate, but right then and there. These students really love that. They want to be real. They want to help; they want to do something meaningful. They want to "make a difference" as the phrase goes and I want to give them those opportunities.

So one of the nice things about the University of Maryland, we just at the edge of Washington DC and we get to work with different government agencies. One of my favourite stories last time I taught a course on information visualisation, students looked at the website of the Bureau of Labor Statistics, which each month produces the jobs report, or the unemployment report; very visible, very influential for the stock market and for every company. What they saw was kind of an old-fashioned design and they built a few versions that had much more interactive exploration that could let you find out how your state was doing compared to another state or how you did compared to last year; all those kind of things became possible. We arranged for the students to get downtown and the Commissioner of the Bureau of Labor Statistics, and her staff of 30 came to that talk, and they listened to what the students did and six months later they changed their website.

Steve Grimwade
Finally, when I'm next faced with a design problem involving humans and a process, what do you want me to think about?

Ben Shneiderman
A lot of things. I want you to think about those people, I want you to talk to those people, that's the lean design and the agile and all these methods are getting out there and talking to people; so that's one. Understand who they are, what they're trying to do and so on.

Then I expect - I tell my students, I expect you to become the world's leading expert on the topic you're working on. I expect you to read every paper that's ever been written about the topic you're working on and that shocks them, but it forces them to realise that what I'm saying is, I want you to choose and focus your attention narrowly enough so that there's just 25 or 50 papers to read and that way, you can know that all that's going on in that area, and you'll know when you've made a contribution.

So immersing yourself in the previous work is an important part; that phrase about "I expect you to become the world's leading expert", think changes their minds a little bit. Students, I think many of them have a weak self-image and I know that half my job is building their sense of self-confidence, self-efficacy and that they should be doing something that works on a planetary basis. Something that's new, something that's important, something that brings benefits to people and then it's not just doing it in the lab or in the classroom, but getting out there trying it with real people and validating that they have something, or if not, fixing it and taking it the next step.

Steve Grimwade
Ben Shneiderman, thank you very much for joining us.

Ben Shneiderman
Thank you.

Chris Hatzis
Thanks to Ben Shneiderman, professor in the Department of Computer Science, and Founding Director of the Human-Computer Interaction Laboratory at the University of Maryland. And thanks to our reporter Steve Grimwade. Thanks also to Frank Vetere from the Melbourne School of Engineering.

Eavesdrop on Experts - stories of inspiration and insights - was made possible by the University of Melbourne. This episode was recorded on December 14, 2017. You'll find a full transcript on the Pursuit website.

Audio engineering by Gavin Nebauer, co-production by Dr Andi Horvath and Silvi Vann-Wall.

Eavesdrop on Experts is licensed under Creative Commons, Copyright 2018, the University of Melbourne.

If you enjoyed this podcast, drop us a little review on iTunes and check out the rest of the episodes in our archive.

I'm Chris Hatzis, producer and editor. Join us again next time for another Eavesdrop on Experts.