Protecting the Humanity of Crowds: Human-Computer Interaction Institute, Carnegie Mellon University

Jeff Bigham, HCII, Carnegie Mellon University

I come from the perspective of having used crowds as part of building systems for the past few years. We have built system that use crowds and computation to drive robots, to answer tens of thousands of visual questions for blind people, to convert speech to text in three or four seconds for deaf and hard of hearing people. We can code hours of digital video in about five minutes. Most recently we were writing an academic paper from a watch, orchestrating a crowd of technical writers behind the scenes.

While I come from this perspective of building stuff, it has led me to talk to a whole bunch of great people who work in this space, so what I did was to tweet out the question: “What are some of the big challenges in crowdsourcing?” I got a lot of great responses and tried to categorize them. There were three big things that came up.

First, how can we teach computers to be as good as experts? While we are all very excited about getting people to contribute to our projects I think that many of us, especially those of us in computer science, are thinking about how we can gradually turn over a lot of that work to computers. I think a lot of the ability to do that comes from how we structure the work that is done by the crowd. It is very valuable, and it is a big first step to collect data that we can use to train machine learning, for instance, but what is the next step? The next big challenges involve figuring out how we can get the crowd to structure a problem so that it is more amenable to machine-learning approaches. How do we get the crowd to figure out what is important about expertise in a particular domain so that the relatively rare experts don’t have to define all of that? This is about cost, but it’s also about speed; it’s potentially about privacy and confidentiality. Once you can get a computer to do it, it’s not some person who is seeing you as data, which you may not want.

Second, what I have been really proud about in my own community and what is really taking off is this idea of how we can build platforms that actually encourage a brighter future for crowd work. The platforms that we build embody certain expectations, certain assumptions about both the crowd workers and also the task that we will have them do. Are those tasks meaningless, menial jobs, or is there something brighter? Is there a path from these little tasks that we ask people to do towards something bigger? You can imagine these kind of crazy ideas like, “Well, I started out as a crowd worker and then I became an expert in the field.” What kind of platforms could you build that could encourage that kind of transition?

My colleague, Aniket Kittur [co-author of “The Future of Crowd Work” available at http://dl.acm.org/citation.cfm?id=2441923] , had this concrete, nice way of saying it: “What would make me proud of my daughter being a crowd worker?” I think it’s a really interesting, compelling question to ask. It turns out Aniket and I both have daughters about the same age, so we are both asking that question.

I teach a class on crowd programming and the first assignment is to have my students, who are some of the top future scientists in the world, make $10 on Mechanical Turk, and it is amazing how much trouble they have doing that. These are the people who are going to go right out of undergrad and make $150,000 a year, or whatever it is. So something seems to be missing there.

This is kind of related to my final challenge, which is: How can we protect the fundamental humanity of crowd workers while simultaneously benefiting from treating crowd workers as computational units? There is this tension that is fundamental for us to resolve going forward. The reason crowdsourcing is so nice is because we don’t have to necessarily always think about each individual as this person who needs to be on-boarded separately, or someone with whom we need to develop social rapport. It’s nice to be able to call people via APIs.

That isn’t great long-term. This has huge consequences for the people who are engaging in our crowd platforms. If we want to make sure this is a long-term thing, something that we can keep doing in the future and really reap all of the benefits from, somehow we need to address this tension. People are both crowd workers and they are individuals, and I think our platforms and the tasks we design need to recognize that.

This presentation was a part of the workshop Engaging the Public: Best Practices for Crowdsourcing Across the Disciplines. See the full report here.