An interview with Larry Yaeger

(Two more high resolution pictures of him using an eMate: one, two.)

This interview was taken by me and Marjeta Doupona at the Worldwide Newton Conference 2009 in Vancouver, Canada. It was transcribed from an audio recording and then edited, so I apologize in advance if it doesn't flow particularly well.


Does there exist anything that does not interest you?
Almost everything interests me, because my background is so varied, partly by choice, partly by blind luck, it just so happened that I was working at a company where I kind of wanted to get out of anyway, and they were going down economically, so I was going to get laid off anyway, and right around that time I heard that these guys were doing a start-up that bought a Cray X-MP supercomputer in their garage, to do the special effects for a film business, and I thought, how cool is that, so I sent off my résumé, and I talked to them - and boom, suddenly I had a different career, then they went completely - they collapsed economically, and the whole company just died, but at the same time I was talking with Alan Kay and Apple Computer and I did some work for them and he said, "Why don't you come work for us?", and so I went and did the interview thing with Apple Computer and I think on a Thursday our company announced that it would go Chapter 11, which means completely bankrupt, and on a Saturday, I got my job offer from Apple, so, I was off to Apple, and I was very lucky to work for Alan Kay at Apple, because he fostered the environment where he just wanted you to do something cool, something interesting; he wanted - preferably - related to the work they were doing, but the whole idea, the project that the ecology in the computer, they just wanted to use it for education purposes, and user interface design, the idea being that if you had a system that was sufficently powerful to model whole ecologies and all the interactions of the agents between and everything it would be a very powerful system, but if you had it being programmed by chidren, from first to sixth grade, it would also be very very simple. And so, that was the driving, the research was for this idea letting children create an ecology in the computer, and I just got interested what happens, and real ecologies and more sophisticated than…
Can you explain this ecology in computers? I've never heard the word 'ecology' used in connection to computers.
Well, using a model of an ecology. So for example, one of the things they talked about was a kelp forest. And a kelp forest is a known interesting whole local ecology where there's certain kinds of fish - the fish used in Nemo…
The orange ones?
Yeah, that actually aren't hurt by the stingers of the anemonies, and there's this symbiotic relationship between them and the idea was that what if we could model the entire kelp forest, create all of these different types of animals that live there, and have their interactions be authentic, be somewhat real, and observe; the children could learn something - the way ecologies work in general; the pieces interact and work with each other or else the whole thing dies and falls apart. And so they could study ecologies - learn by doing. They could study ecology by creating one; they could model it in the computer. They designed the behavior of the fish, they designed the behavior of the anemony, they designed the behavior of the crabs.
In what way? How did they design, for example, the behavior?
They write software.
They write software?
First to sixth grade.
First to sixth grade?
Six to twelve years old. But the software that they were gonna write wasn't normal programming. Alan was looking for a way to make it possible for children of that age to begin building their own worlds, building their own agents, and the behavior of these agents. So that it's easy enough for children to understand and do and make these things and process and learn these kinds of systems.
But this project never came alive, or…
Actually, it's gone through many iterations, I think the first version was called Playground.
How many years ago?
This would be about 1989 or so.
Now we are in 2009 and I don't know any country in the world where children use computers as a regular tool in classrooms.
Actually, Alan's latest project - I think the current incarnation of it is called Squeak, and it again is…
It's the one you talked about today?
Yes, and again it is a programming environment/language for children to do interesting things and people are putting it to some use, also…
What do you mean by 'people'? I often find people reluctant to the idea of children using a computer in class; they say they don't need them.
(sigh)
You know?
Now, there is one key thing I think Alan was smart to do, which was the purpose of having the computer in the class was to never teach the children about computers. The idea was always about having computers facilitate education about whatever other things they were going to learn anyway. So, we worked in Los Angeles, with a so-called "magnet school" that attracted…
It's kind of like a model school?
It's kind of like a model school that was supported by the state, but it had a special program, it was sort of an experimental school, and I think students had to apply to be in it but I honestly don't know what the criteria was, and they had very very good teachers. You know you have a good teacher when it almost doesn't matter what materials they use, so they had very good teachers. But we also put computers for like every student and help them develop lesson plans, they really took HyperCard… I don't know if you remember HyperCard but it's this programming environment that is kind of like using index cards that can tell you to go to other index cards and make things happen on the index cards. And it had hyperliks to other cards, and Bill Atkinson, I think, is the primary force behind that, and the teachers who didn't consider themselves programmers found that they could make HyperCard present lessons in interesting, fun ways to have students interact with it and help them learn whatever subject they were studying - not computers, not software, not programming, but use this as a learning aid for whatever…
Like pencil and paper…
Yeah, just like one more tool.
But this was that school in Los Angeles?
Yes, and I don't know who's teaching Squeak currently, but I did have here my computer didn't I? (looks around) Oh, here it is. Squeak is on the web and I bet we can find some examples of people who are actually using it. There's a fellow at MIT, Mitch Resnick, who developed this language… Oh for heaven's sake, I hate my memory, it comes and goes. (laughs) It's based on the idea of turtles, and I got it on my computer, you can do graphics, and having the turtles move. So you say, move a few steps forward…
Isn't that Logo?
Logo, that's it. Logo and Net Logo, that's it. Thank you. And he has found out that you can start with very young students, and they do some really neat projects with that. One of my favorites is the sort with termite mounds, you know this?
No…
It's kind of like a neat artificial life type thing. They give these little agents rules, that is, basically, "Run around randomly until you hit a lone piece of material", I think it is a wood chip, and then, "pick it up. Once you have a wood chip in your posession, run around randomly until you hit more wood chips. And when you do that, drop it.", and that's it, no other rules. They take this and scatter wood chips all over the environment, these termites all around the environment, and they'll run around, and what will they do is that they will aggregate wood chips and do a pile. Sometimes you would get two or three piles, but even then, they'll sort of gradually ones from the smaller ones and if you let it run long enough they'll always end up with one bigger pile, with sort of no cooperation. No communication. Nothing but this individual low-level behavior - pick it up if it's by itself, and put it down if it's more than one. Done. And it's good for children to see and understand the way group dynamics can play out and how organisms can cooperate without even cooperating.
But we are still, as a world, very far from that computer environment classrooms. I think that South Korea might be the furthest of all because they are planning to have all textbooks and everything on computers by 2013, so everybody would just use computer based textbooks. No more paper. But every other country, including the USA and all European countries, is very far away.
I think you're right. I think Amazon is trying to get more and more electronic textbooks available on Kindle, I'm hoping that these rumors we're hearing about a device from Apple are legitimate and they will have a really nice color, decent sized screen, available with PDFs, and do all the things the Kindle can and the Kindle can't. Maybe these will come as good book readers. Still, they won't be cheap, textbooks aren't cheap, so…
I think it pays off in one year. I compared Slovenian prices to one regular notebook computer, and it's about the same. But, you see, in Slovenia teachers still think that a computer is something that is used as a reward or a punishment - if you are a good student, then you can use a computer, and if you do not behave well, then you will not use a computer. So, maybe that's because some children are capable of using computers more than teachers are. The average age of teachers is about 40, and most of them started to study in university after personal computers were introduced - they missed that period, and now they don't want to know about it. They know how to Google something, to use e-mail…
They know how to use it, but they don't know how to make it do what they want it to do. But they don't even know what they want it to do, because they're stuck having that experience, but that will change. Nowadays, it's a cliché that the adult has to go to the teenage son to figure out how to get the computer do something. And as the children grow up with this technology, move in to the educational system and become teachers, they'll know how to deal with it.
Yes, that's right. I noticed today that you have a lot of electronic equipment with you; how much of it do you have?
(laughs)
That you use regularly, what do you take with you?
Well, I almost never go anywhere without my laptop, my MacBook Pro. These days, for the past year or two, I've never gone anywhere without my Kindle, my electronic book reader. Either my wife or I has a digital camera with us almost always. I always have my iPhone.
Do you have a Newton?
I do have a Newton, but I don't use it regularly. I love the operating system on it and I want something like it a little bit smaller. I want the iPhone to have pen input. And then I would be completely happy with it… Steve Jobs has this thing against pens. (laughs)
You said something very interesting today. I found this conversation on NewtonTalk why the Newton was cancelled, and everybody said that was just because he didn't like it, but today you said that he drew very capable people from that project on to another part of the company in order to save the company. So that's something really new.
He has said that, I think in print. I'm not making that up. Some people don't believe him, some people always believe the worst thing, they think he killed it just because it was Sculley's project. I really don't believe that, but that probably contributed to it. I believe what he says, that he did that to save the company. He really single-handedly turned the company around. I joined Apple in 1987 which was almost exactly when Sculley forced Jobs out, and I saw the Sculley era, and there was another guy who ran the company for a while. I can't remember his name, (Michael Spindler?) and there was Gil Amelio, and I watched the company just get worse and worse and worse and worse. There were for a long time rumors of Apple's death, I thought it was just silly, the company was doing pretty darn well, but towards the end of that period we really were losing money and sales were falling and it was clear there was no leadership on top. I thought, finally, right towards the end that we really were in trouble as a company and then Jobs came back and truly turned it around.
How quick did that happen? I mean, from the time you were regularly working on it and until the day they said it's cancelled.
The Newton?
Yes.
(starts thinking heavily)
Just approximately so I can imagine how things happened.
The actual announcement was a little bit of a shock to us. Because the company had been spun out, it was going to be its own independent company and then Jobs came back and said, "No, we're going to spin you back in and you're going to be part of Apple." Not cancelled, but part of Apple. And then, I think it was only a few more months before it came down that it was going to be closed. So, the whole process of Jobs coming back, being spun back in, and being cancelled was less than a year, I think.
People in the Newton community are now afraid of that 2010 bug
Right…
So, I don't know how you have been following NewtonTalk…
A little bit. I don't read everything, because it's just a little bit too much for me.
Yes, there's a lot of it. But people were having an idea to approach Apple for the source code, so they could somehow get over that obstacle, so they could reprogram Newtons and they would work. Do you think that something like that could happen? My question goes to the field of copyright. I come from an ex-socialist country, and we are not that aware of copyright because we don't really believe in that. I know that also, in theory, there are different approaches to that, what is common, what is human knowledge and what is personal knowledge, and what is personal advantage of that and everything, but I just want you to comment on that, because you are the author of very capable software. How do you look on that? Do you think that Apple would be willing to release something?
Personally, I wish they would release it, I wish they would make the source code open source. That would be my choice.
Why?
Because they could retain copyright even though they would make it open source. And there's different ways of making things open source; you could make it so you could do anything with it that you want, you could make it so you could do anything you want but mail the changes back, make it so that you could do anything but it doesn't make a profit, or you could make it so that you can get a profit, but we get a percentage of that, there's so many ways they could do it, but at the minimum, if they made the source code available, at least there would be a chance that something, you know Paul, the guy who did Einstein, you know, he would have had so much of an easier job if he had been able to get to the source code, and there are people seriously motivated, like Eckhart, that would probably have Newton software running on all kinds of devices, if they could just make it available. I wish with all of my heart they would do that. However, do I think they will do it? No. I think that they are still bound up in the intellectual property, it's a little too much, and that they just don't think that they should do it. And also if, if, they are coming out with a pen-based computer, they may also be feeling they are giving away a key part of their current business, in which case they would be even less likely to do it. Also, as I mentioned today, there is a problem that they don't own the rights to the cursive recognizer, that we only ever received just executables; binaries. And so they couldn't give that source code out, because they don't have it.
And who has it? Is it the Russian…
Paragraph, and that company probably isn't even called Paragraph anymore.
Today we talked about tablets, and there was a big debate why to use tablets. For example, some people almost forgot their handwriting; they hardly sign themselves, because they just use a keyboard. So, why do you think that computers should learn how to read human writing?
Well, it's just, like we were talking today, in situations where a keyboard's just not convenient or appropriate. Sitting in meetings, people use them but they just make too much noise, it's kind of rude, whereas you can take notes quietly. Sometimes, you have to work in a standing position, like UPS, or doctors in hospitals, insurance forms, there's all kinds of situations where typing is inconvenient whereas if you had something that you could put in the crook of your arm or hold it in your hand, you could take notes for whatever you need to do, and it's just a more convenient way to interact with the thing.
Do you think that people in general prefer to write by hand?
It depends. It's totally situation dependent. In a note-taking situation, it would better to have the pen, especially if you could do a mix of text and graphics, you would draw things, you know, professors stand up at the board and draw pictures, and it's kind of hard to do that with a keyboard, whereas you could draw the same picture if you had the pen.
This is an academic situation and environment. You know both worlds; academic as a scientist and as an industry. My impression is that people who work in an academic environment just forget about everybody else. Because they don't know anyone who would be more different than themselves, you know, they're well educated, more or less well paid, they also have educated children, a lot of resources at home, but what about the rest 90% of the people? What do they need? I assume that industries have to respect others because they want to sell things to everybody. Did you start as a scientist in the first place and then came to Apple or it was just… you said it was coincidence, but…
What I studied in university was aerospace and engineering.
Yeah, I've read about that. Why is that?
You know, looking back, I didn't know this at the time, but there were probably two things: I loved everything to do with outer space. I read most of the science fiction they had at the elementary school library, as of the sixth grade, I read all of the science fiction. I was into space. But also, my father worked for Pan-American Airlines, they don't even exist anymore, but he used to work in the parts area where they would repair and maintain the airplanes, and to him, the coolest people in the world were the guys who designed these airplanes, and so I went to become an airplane designer. So, I didn't realize that until much later. I'm sure that influenced me. And so my own love of outer space and this connection to my father's thinking, it was so cool to do. It was natural for me to do aerospace engineering. I never really considered anything else.
Did you ever construct an airplane?
I designed one, and built a model, and tested it in the windtunnel, but I never actually built a full-functioning airplane.
A model?
The model, which performed well in the tunnel. (laughs) And I wrote software to analyze how it ought to perform in the windtunnel and it performed sort of like it.
I want to know how you came to handwriting recognition. Because it's not so obvious that from constructing planes you came to that.
Well actually that was a combination of things. I went to work for Apple, we worked in the Vivarium program, and we started artificial life research there.
Yeah, you explained in the beginning how you were drawn to it.
I was modelling ecologies in the computer, and one of the key technologies there was neural networks. I wanted to design something that behaved like the human brain behaved.
So you started biology in your free time?
Well, Alan was very good at it, he supported me to go to the first ever neural network conference, the first ever artificial life conference, so I studied all these things by going there and meeting the right people and hearing the right talks and reading the right papers and stuff, and worked on that for a couple of years. I presented a paper on it and artificial life at the conference, and right around that time Apple was starting to realize, maybe the budgets aren't always going to get bigger, they're starting to shrink now. And they were looking around to see where we were spending money that is not making us money back and I was one of those things. They said, "Is this going to turn into a product in the next two or three years?", and I said, "No, not by chance.". And they said, "Could you please go to something that we could make money off of?", and, it was still research, and then I worked on handwriting recognition, because at least I can keep up my neural network interest and use that as the core of the thing, and it actually may be useful some day. And so I hooked up with a couple of guys and we worked on it, and one of those guys actually stuck around, and we put in Brandon Webb, we put a lot of time on it and had more people coming in on it, and it worked, we were good or lucky or both and it worked, and everything we chose to do paid off, and right around that time, turns out somebody did need it, the Newton needed it and PenLite would have needed it if would have gotten out the door and so the technology actually shipped. But because we were doing a lot of things that were sort of vaguely biologically related to the Vivarium program including neural networks, and I understood how to use those, I'll go apply them over here, in something that will actually ship.
It's not that obvious that handwriting recognition is an artificial intelligence. It is kind of, but it appears as something more or less technically related. You have an algorithm and you can capture all handwriting.
Well, when we went to do French and German, we actually went to France and Germany to get data because people write differently. The number 1 in France has a big long loop on the front before it goes down. No one does that in the States. We wouldn't have a single example of that if we only used our US data.
How could we approach spelling errors? When a human is unsure about what is written, they can help themselves out by reading aloud. So this is probably the next step forward, in that the computer would try to guess the word by itself. Maybe it would be even better than humans, because it wouldn't have to wait for the moment when it would cross one's mind, "Oh, I should read that aloud.". And the computer would just do that automatically. Is reading aloud possible for computers too?
Reading aloud gives you an extra 'sensory channel' that the computer doesn't have, unless the computer could try converting it to phonemes, and effectively 'read it aloud' alone all the time, and maybe that would be even a useful way to do recognition under situations like that when there are spelling errors.
For example, yesterday at the laundry, I saw a notice "I need a job", where "I need" was spelt as, "I nedd". But it was obvious what it meant from context. But would it be obvious for a Newton, if I tried to enter it like that?
Probably not. It would probably put "nedd".
I am interested in how you made contexts. How did you do that?
I will get into this tomorrow, but our context is only at the word level. Or at least pairs of words. But really just words. There's no sentence level.
But is it structured, like sentence syntax?
No, it's just words. So, I always thought, that another step in improving it would be to have something that had at least like phrase level context. And some of our variable-linked Markov models are very good at that, what they do is they look for sequences of characters, but sequences of characters including spaces, and therefore sequences of words. And you could conceivably have a much more effective language model. Because it would have some sense that "I need a job" is much more meaningful than "I nedd a job", where as just looking at "I", and this thing "nedd", well, if they wrote it carefully, we would say they wrote "nedd". And then "job" all by itself. But if it was looking at the whole phrase, then it maybe could actually correct it. But you would have to be careful. Every time you started applying correction, well, people write some of the strangest things and they want it presented that way. (laughs)
How far are computers in capturing the human language in general?
For recognition, or…
How could a computer understand, for example, like in movies…
2001? "Open the pod bay door, HAL…" (laughs)
And when computers talk to humans, when they make sensible sentences, how capable is a computer of simulating a human language?
It's actually pretty good at making the sounds, it's actually pretty good at speaking words but it's not good at understanding any of that. That's why phone trees, I don't know if you have to deal with phone trees as much as we do, phone trees are, you call up some business and you just want to tell them you've received the wrong product, that they've sent you something else. So you start and it asks you, "Which of the following are you calling about? Sales, technical support, blah blah blah blah blah blah," and it goes on and on and on, and your thing isn't any of these. And so they say, "Do you want to know about store hours, do you want to know about directions, do you want to know about price, do you want to know about…", and it's so structured and it's so completely lacking in understanding, that you become very frustrated.
It's pretty much the same as when you have a problem on your computer, and the computer asks you what kind of problem you have, and it's never on the list.
You're right… (laughs) The thing you actually need is never there. (laughs)
I could reproduce what you said today, but I would just ask you to shortly explain again, what is the main problem of handwriting recognition. When you try to make software for that, you were explaining everything but I cannot reproduce that for the sake of the interview.
Well, I'm not sure what you would say is the single biggest problem…
No, not just one problem, you can mention like ten of them, like slashes and "I"s and "L"s and, you know.
There certainly is the confusability of individual characters. That's one big recurring theme. Even with one person, you might have a hard time distinguishing between an L and the number 1, or a lowercase L and an uppercase I if they don't put nice crossbars on it. And there is no difference between a lower case L and a vertical bar except maybe the height, and that's only a clue if you sort of know to draw the vertical bar a little extra taller, so there's all this confusability about individual characters. You know, if you draw this, is this an O or a zero. You cannot possibly know.
I ask you because I think that most people would think that this is something really easy.
(laughs) Handwriting recognition?
Well, you just put it there and the computer will read it! Like if you have a big database on the computer, some people just think that you hit Enter and all the data and the analysis, everything about statistics, even computated models, just run out.
Well, I spent twelve years of my life doing that stuff and it's not easy. (laughs)
And you found it interesting all those twelve years?
Oh, the first part, the first four years were very interesting.
Did you think it would be easy to do that when you started?
No, I didn't expect it to be easy. I wasn't even sure that we would succeed. We just thought that we would try, give our best shot, and like I said, I almost felt lucky that the path we chose actually worked. But there are so many ways. But even, like I said, when we got the basic structure engine in place, it sort of worked, but didn't work good enough. And then I spent a lot of time analyzing errors and finding out what went wrong, why it went wrong, and how to change something in the search algorithm or in the representation of the numeric, or modulating what the neural network learned in order to make it do better and not fail this way. And try to never do something specific to one thing, but solve the class of problems this represented. So you rule out a whole set of errors. And then you find another error and you try to understand it, you try to generalize it, you try to find out what caused it, and fix that whole class of errors. And that process went for several years. That took it to make it work really good and make it useful and that you could depend on it.
But it's still there in Macintosh computers (as Inkwell). So how do you feel about that?
Oh. I'm really happy it's there.
Do you consider it a piece of your work or how do you emotionally react to that fact that it is there?
Well I'm proud… I was upset when they asked us to remove the easter eggs. (laughs)
Easter eggs are great. I have a few questions on that.
You know, there's more than just the "Rosetta! Rosetta! Rosetta!" egg. There are quite a few of them, actually.
There's someting about "Larry" too, right?
Yeah, if you write "larryy larryy larryy"… what did it have to say? I think two different versions say two different things. I think one said "The Doctor is on.", which is a reference to a fire science theatre line. And another one says… I can't remember what it says. I'll have to try and see.
I read somewhere it said "Good Worker".
Oh… I think it says "Hello, Worker!", which is another fire science theatre line. (laughs)
Some of them I found and some of them I just read of. I once read a list of them, I couldn't get any of them except "Rosetta! Rosetta! Rosetta!" recognized. (takes eMate out of backpack)
My wife used an eMate for a long time. She just likes the sleekness and the size, and she didn't need a whole lot of capabilities, she just used it as her text editor and would type and write on it.
I got one Newton repaired once by Frank Gründel in Germany…
I met Frank at one of these Newton conferences, he's such a nice guy, he's really cool.
Do you know about computers from the OLPC project? What do you think about them?
I actually got one, I made my donation so they sent one and I never had too much luck with it; it was too slow, so now it's just a pretty paperweight. But I know people who made much more use of them, so I just didn't spend enough time with it, I think. Have you used one?
No, I was trying to buy one on eBay, but I was close only once and I was beaten in the last second so I couldn't buy it. I would just like to see how it works.
Ah. Let me write onto it (the eMate) and see if I can remember all these…
So why are there easter eggs in software?
I don't know, programmers have been doing them forever.
So it is like a habit?
Yes, it's just that people who write software have control over the things so they like to put little things in it.
Like pictures of programmers?
Yes, and Breakout games, one of the earliest iPods had a Breakout game that nobody knew about unless you knew the secret way to get into it. […] Let's do the "larryy larryy larryy" one… […] Larryy larryy larryy, The Doctor is On! (laughs) OK.
Maybe you have the privilege that it knows your handwriting better than anyone else's?
And this has trained me. I've learned too. So let's see… I think Brandon has one on here, so "brandon brandon brandon", let's see if it gets this right… I guess not. Maybe it has to be uppercase. […] Yes, it turns the last one into his address, brandon@brainstorm.com. (not sure whether this is correct) I think I put mine in without the extra Y.
There were many programmers on this software. Did they hide these easter eggs one from another?
I put all these in. […] There we go, it turns one of these into larryy@apple.com. I think Dick Lyon had one. I don't remember if it was "dick dick dick" or "lyon lyon lyon". I think it's "lyon lyon lyon". […] Nothing. So that wasn't it.
It's a kind of signature.
Exactly; it's a way of signing your work. […] I would have sworn I had one in there for him. I can't remember what it is I guess. I don't think this generation has that one or not, a place in Italy named like the restaurant where we ate way too much.
But the handwriting recognition, is it now yours, or…?
No, it belongs to Apple.
You keep your moral rights, is it so?
No, the company owns everything. I would have liked to start a business using the software and I tried to licence it but they wouldn't licence it to me. […] That must not be that version of software because Mondello isn't even in the dictionary. That's why I'm having so much trouble with that word.
Do you have any idea why the eMate only has printed recognition and not also the cursive one?
They simply didn't want to pay for the license. Honestly I think the main reason was a business decision; they didn't want to pay for the license for another version of the Newton. (continues writing)
But you find it amusing, don't you?
Yes. (smiles) Yeah, "rosetta rosetta Hey, that's me!".
This one is great, really.
Let's see if this one is in there… OK. There are other ones in there I just remember what they all are. I got Brandon, I got mine, two for me, and Rosetta, I had one there for Dick Lyon, I just can't remember what it would be; if it's not his first or last name… maybe I did it lowercase… I think I did that…
But you did say today that Apple contacted you again about the recognizer?
Yeah, just recently. Oh, there we go, I put "lyon lyon", and Richard came out on the third one. So that's his first name. Anyway… The most recent contact was just about the patent filing with this going on. And it was about a year and a half ago or so and they asked if I knew anyone who would be interested in doing anything with the handwriting recognition.
You are probably your own boss now at the university. And I got the impression just by reading your page that when you want to learn something new for yourself, you announce that you're teaching a new course. Is that right?
(laughs) You could just about do that. It's not entirely freeform. For example we needed someone to teach introduction to computer programming, and I really like programming, so I was kind of a natural choice for that. But there's another professor and I've been talking to him about the evolution of ethics where there's been people doing studies of simple prisoners' dilemma problems and the best rule that comes out of it is where you do what the other guy did last time and I think if you put a system like that together in the right kind of environment where populations can interact with each other I bet you could get it to evolve… in other words, cooperate first. So it's kind of the inverse of the golden rule, do unto others as they have just done unto you, whereas if you could show that if you evolved a strategy of do unto others as you would like them to do unto you, I think that would be a pretty good evidence of the evolution of ethics. The golden rule is more of moral, ethical behaviors you could adopt…
But there are two approaches in biology to that problem, aren't there? What do biologists say about your approach?
Well, this is pretty far removed from me.
So they don't interfere with your work?
Well that one with the prisoners is a common problem and has been used all around.
So when people come to study to your class, are they people who want to have their hobby as a job? I mean, this is all very interesting and you don't know where it ends or where you finish your degree, what kind of knowledge you will have. You just study because you enjoy. I saw a list of your courses, or, no, the titles of works that you demanded from students; they were all interesting works, I have no idea whatsoever what this could be, using not only your knowledge but also your imagination… you know, everything that comes with "imagine another form of life or another perspective"…
The course you're talking about - artificial life and artificial intelligence - is deliberately geared to expose the students to a lot of different ways of thinking about things. There are several seminal papers that heavily influenced me and yes, I want them to think about life and intelligence as a process, as a dynamical system, and I want them to understand the way that dynamical systems behave when they can evolve over long time periods, and I want them to have the ability to think about how do you assess what's going on inside that system using information theory and how do you measure the complexity of these dynamics, how you put it together, and you try to leave them down to this path of thinking the way I do about life and intelligence and evolution so in the end, I hope some of them will go off and do much better than everything I've ever been able to do and if more people were doing what I'm doing, I retire. (laughs)
Is there a definition of a good professor?
Yeah, you really want people to run with the idea and improve upon them.
And then artists come to approach you or are you also involved in movies, I mean there are all kinds of stuff that we could talk about that you worked on. I mean; isn't that too much for one person? How can you manage to do it all?
There's not enough time. There really isn't. There are some things that I would love to do. A friend, Norbert Herbert, I did a project for him that I did just because it was fun. It was just an art project, it wasn't a science project. I used a flocking algorithm which are very well understood. And there was this friend who invented the algorithm and I took his software and just adapted it for our purposes and we made a system where you would have an enclosed space, and we put black stuff on all the walls, black stuff on the carpet down, dark ceiling, and we put pillows around the floor, and you would lie on the floor and look up, and we put a special projector that would fill that space and would show these flying flocking artificial birds going around and I made some invisible things that they would interact with and it would infect the flock dynamics and then wrote special software to extract aspects of their motion and turn them into sound and you could see who was producing the sound. They would produce these long, swooping, low-pitched sounds and once they were all clumped up together they would make these high-pitched sounds that were short-lived and it served no purpose, but people liked it; people spent a lot of time in that environment. My theory why it attracted people was that the motion was very organic and very real and natural, and yet the birds themselves were just polygons. It didn't even remotely look like a bird, it was obviously artificial. And that blend of the artificial behaving very naturally is something that you see and it's amazing because it's not one or the other; it's both. I've got a hundred ideas for things that I'd like to do along the direction of art projects.
But if you have a big idea, can you divide it into pieces and then just give students piece by piece?
I've got a page full of artificial life ideas that I could give to students as a project for the semester, I show them a list of all these ideas and say, "You don't have to pick any of these - these are all recommendations; you can pick your own right from scratch, but here's some suggestions."
What are the best ideas they had and you never imagined? Something that surprised you or…
There was one guy who worked on a project that studied numeracy, the ability to understand and use numbers. He was looking at the very first basic principles how there evolved systems that paid some attention to counts of things, and that was kind of interesting. There was another guy who was extending work he was already doing with another professor, Randy Beard. Randy already had these cool agents who ran around the bottom of the screen, and you'd drop things on them and they used neural networks; you could train them, you could evolve them, to for example, avoid diamonds and run towards small spheres, or, catch small spheres and avoid big spheres. And he worked on another thing that was about communication where you had an agent out here that couldn't see, but could effectively hear, it could get a signal from one that would sit over here, and could see. So they evolved a sort of communication for one to guide the other one and do the same task. […]
If you work on a project of artificial intelligence, is this human intelligence?
Well…
Not necessarily?
Not necessarily. Because in fact, I like to think that intelligence is an entire spectrum from the simplest single-cell organisms developing to an upper chemical gradient is a kind of intelligence. It's not much like ours. But it's a useful adaptive behavior to the environment to survive and reproduce. I think anything like that is reasonably considered to be intelligence. Now if you take that perspective, then you see this tremendous increase in intelligence over evolutionary time scales. And what I'm trying to do is start simple, for example with a computational worm, before I evolve into a seaslug, before I evolve into something else, before I even think about something like human intelligence. But along the way, there should be some useful milestones. The point of this is to show interesting behaviors. So that's my take on this; yes, sure, I want to hit human intelligence, I want to go beyond it, but I'm not going to start there.
Do you think that common perception of artificial intelligence is that somebody thinks of something that is less than human?
I'm not the only one who thinks this way. In fact, I can point to Darwin, there's a quote by him that I show to my students all the time, "If you only imagined a human level intelligence of a whole different nature…", I can't really remember this part… "then you couldn't understand how it evolved; there would be no way to say that it came along gradually through a series of evolutionary steps. But in fact, if you look around you, that's just not the way it is, there are all these levels of intelligence." The way I think is that there are all these levels of intelligence, and that's why we are where we are today.
Does that mean that survival of the fittest includes that kind of intelligence you're talking about? There's a lot of stuff on Darwin's theory in recent years. I think it's really interesting because survival of the fittest was told in many schools as survival of the strongest which is not necessarily… it just comes to point of reproduction and then you are useless when the species is removed… like there's sickness, and aging… I think there are many interesting books on that subject, how to incorporate everything into survival of the fittest. So I think it's really an ambitious project to incorporate everything…
Natural selection and evolution are just a complete topology, and it's basically that damage survives… you know, there's nothing that you can even argue with. It's just if it makes more of itself, there's more of it around. Usually I start off by telling students it's a topology. Like, "well okay, there's more than one," and it happens to be a reproduction so I say, "it's also that which reproduces - increases its numbers". Well, okay, you can't really argue with that, you make more, there's more there. And then OK, one more thing: things change. For example, you're faded by another organism, the climate changes, an asteroid hits… things change all the time, and so the environment that you're having to adapt to changes all the time. It's not ever good enough to be completely static. Although cockroaches are pretty good at it. They've got a well-developed strategy and don't have to care what's going on. But basically, evolution is this topology: that one which survives persists, that one which reproduces increases its numbers, things change. It's lovely to me that something that simple and elegant is amazing… it's magic. I don't need to invoke creation myths or…
But there are some states in the US that abandoned Darwinism…
Yeah, Ohio… no, wait, it's not Ohio. Sorry Ohio! It was Kansas that tried to do that for a while and Texas did that for a while. So far every time a school full of religious zealots, they've been voted out and it's been fixed. I don't think there are any that keep doing that. They keep trying. Their religious beliefs are their own business. I don't want to go around telling anybody what to think or believe. But when they start using their beliefs to tell me, and my children, how to think and live, then I'm angry. Most organized religions are dangreous, I think. They're so wrongheaded that they do damage and it's a shame. If they just believed the way they wanted to believe and not try to tell people how to act and behave, that would be fine. But they don't do that. There are movements in the US that try to remove the separation between the Church and state. To make it a religious state. And that just scares me. I wouldn't do anything like that.
I don't want to take too much of your time, but I'm really interested in your experience with that gorilla.
Oh, Koko.
Yeah, because I know a lot about it, but I didn't know until I read on your page… I was reading everything the sign language that she used… and what did you do, some kind of software, that she could touch on the screen, and then you could hear something?
I had the easy part.
How did you get engaged in there in the first place?
I was still working for Alan Kay, and Alan has this adorable tendency to basically put everyone on the board of directors that he wanted to have dinner with. So that's how I got to meet Douglas Adams…
The famous writer?
Yeah. And he was interested in Koko and all that, and so he invited… technically, Koko the gorilla was on the board of directors. So he'd go hang out with them. I'm not sure who initiated it, I think the people of The Gorilla Foundation had a system in the past that had a big keyboard with keys that a picture on it and she could press them and I think it played back sampled sounds and words.
I think I saw that on TV. That screen. But I don't remember any sound connected to that.
I don't remember much about this, but I think it was too easy to damage, you know, she'd smash the keyboard, she'd step on it, and it stopped working fairly quickly so they didn't get a whole lot of use out of it. So like I said, I had the easy part, but these other guys in the project had to design and build the hardware. And Koko, she would get up and sit on it, and bounce, and she wouldn't just press with her fingers, she would use her whole knuckle, and other things, like stones and stuff, and slide that across the screen, so it was hard to keep this thing functional. But the guys did a great job, they built this strong plexiglass surround for it and a touchscreen that had this finish with diamond coating, and made it actually work and stay working. And I wrote the software. She had a cloth with pictures and words, and she learned to associate those pictures with the words, so I scanned it in into a database with the collection of images, and The Gorilla Foundation had recorded different people speaking and they asked Koko which one she liked. And Koko picked this one particular voice. So they had the speaker record all the words that corresponded to the pictures and I put those in the database and made it that the software could be put into a sort of "editing mode" that the humans used and it would lay the screen out and associate pictures with buttons and sounds and then they would take it out of the editing mode and give it to Koko and she could walk up on the display and she would see those pictures that she was used to and she could touch it and it would play this person speaking.
Like what?
You know… apple, banana, nut, lip… and some words had different meanings, for example "lip" also meant "girl".
Did she combine them?
I wrote it deliberately so the software in the background recorded a time-date-stamped record of everything she entered so that you could tell what she wanted to tell. Maybe they did some linguistic analysis on it. I always thought they would have this archive now of all her spoken language and it would be really interesting to look at that data, I think. But I don't think it ever happened.
Did you ever try for example, if you changed the sound of say, "banana" and it would say "apple"?
(laughs) She recognized spoken language. I think I've seen her respond to spoken English. But she was usually doing sign language.
So she used two systems to communicate with people?
American sign language and the computer. I got to interact with her a little bit - I don't know the sign language, and I made a silly mistake of… I mean, I know she couldn't understand the concept of a computer company. But we were from Apple, so I learned the word for "apple" and when I got there I said something like "apple over the weekend", and when I told her we were from Apple, she immediately went to look in everyone's bag or backpack, looking for an apple. And she also wanted to look in all of our teeth and one guy had some gold fillings… a gold cap, that's it. That was her favorite.
Do you have any animals at home?
Yeah. Dogs, cats, and horses.
What's your opinion on piracy? A lot of software is still pirated in Europe, it's the norm, and also music and movies.
Hmmm.
So… that's a kind of habit. I don't think, when I was saying about intellectual rights, that you should use something that somebody else did, and you could take advantage of that. I don't like that. But I don't know what's wrong if you just use it and see it for yourself.
I understand that somebody who spends millions of dollars making a movie wants to get their money back. And I can understand them. I want them to get their money back because I want them to go make another good movie. But there's a limit to how much they deserve to make back and it's not like someone in Slovenia who wants to see a movie and it's not even showing in the cinemas there and you can't… if you would pay legitimately it would cost so much that nobody would buy it. Two people would buy it. A few rich people would buy it. The person that downloads this and sees it and learns about all the people involved in it, maybe some day when they have money, they'll pay for a movie that involves those people and you can think of that as advertising for the next movie, but they didn't lose any money, because that person couldn't have paid for it anyway. They simply wouldn't have experienced it if they hadn't downloaded it. Whereas if they download it, they see it, they learn who the people are, and eventually it comes back around, they'll pay for it. When they can. And I have no problem with that whatsoever. I want to support them, and I want that the people who make these things keep making them, but I also think they're crazy to think that they can get every person everywhere to pay a full price for these things. I'm tired of it. I bought the same exact movie on videotape, DVD, and now I'll have to pay again to get it on Blu-Ray. I'm tired of this now. (laughs)
First published on .
Last updated on .

Table of contents

Contact me