## TRIP REPORT HCOMP-16 Austin, Texas, USA Oct, 30 - Nov, 3 #### [Prakhar](https://prakhar4.github.io/) writes about his trip to Austin, Texas, where he presented a paper at HCOMP ### About: [HCOMP](http://www.humancomputation.com/2016/) attracts researchers who are broadly excited about the fields of Crowdsourcing and Human Computation. After three amazing years since its genesis, 2016 had its fourth edition held at AT&T Conference Center, Austin, Texas, USA. Five days of HCOMP-16 were packed with exciting keynotes, talks, workshops, tutorials, poster sessions and demonstrations. It was a great learning experience to interact with researchers at the frontier of crowdsourcing and understand their perspectives on relevant problems. The participants ranged from a wide spectrum of behaviour scientists, psychologists, designers, educationists, mathematicians, experts on security-privacy, human perception etc. Attending the conference during halloween and its subsequent week made the trip even more enjoyable. ### Keynotes: [Iyad Rahwan (MIT)](http://www.mit.edu/~irahwan/) addressed the first keynote titled as “Extreme Crowdsourcing”. The speaker highlighted his extensive experience of participating in several DARPA challenges that required quick action plan on real crowd workers and presented their rigorous analysis that were conducted after the competitions. Iyad spoke about their approach for DARPA Red Balloon challenge, Tag Me challenge, Shredded document reconstruction challenge. One interesting aspect of his talks was the adversarial attacks from competing teams. Adversaries impersonating volunteers to confuse the judges, launching network attacks etc. underscored the requirement of robust mechanisms. Second keynote by [Ashish Goel (Stanford)](http://www.stanford.edu/~ashishg/) on “Decision Making at Scale” was centered around eliciting public opinion at large scale to make democratic decisions. He presented knapsack based algorithms for budgeted voting and their real-world implementations. The fact that preference of people change when budget is exposed to them was emphasized. For example, asking local public to choose if they would like the Govt. to refurbish the roads or construct a hospital, will have different results when they are told that cost for refurbishing roads is $10,000 against hospital construction for $1million, given that Govt. has upper bound on budget. Third keynote by [Nathan Schneider (Univ Colorado Boulder)](http://www.colorado.edu/cmci/people/media-studies/nathan-schneider) was on “Internet of Ownership”. The speaker brought forth designs of ownership which could avoid monopoly or labour abuse. He spoke about ‘platform cooperativism’ to build environments that give the power of governance to people who are impacted directly. Instances of Green-Taxi, writer-journals etc., were given as successful instances power sharing. ### Panel: Will AI run the show? What’s left for Crowd platforms? The panel of experts speculated the future of crowdsourcing along with the fast paced development of Artificial intelligence in this session. They reasoned that several tasks that require human intelligence today will be machine-solvable soon (in next 3-5 years). However, the panel concluded that human intervention would only help improve such systems and still be widely relevant. ### A few interesting papers: **[Efficient techniques for crowdsourced top-k lists](http://aaai.org/ocs/index.php/HCOMP/HCOMP16/paper/view/14036)** (_Best paper finalist_) _Luca de Alfaro, Vassilis Polychronopoulos, Neoklis Polyzotis, UC Santa Cruz_ This work proposed techniques to obtain top-k lists out of larger sets using human comparisons. In particular I liked their strict assumptions which took care of constraints such as the total number of tasks need to be very few due to monetary reasons; human workers may have high latency; tasks posted should be smaller in size; workers may disagree etc. Their randomized algorithm could save budget for very large lists. **[Click Carving: Segmenting Objects in Video with Point Clicks](http://aaai.org/ocs/index.php/HCOMP/HCOMP16/paper/view/14045)** _Suyog Dutt Jain and Kristen Grauman, University of Texas at Austin_ This work aimed to build an interactive video object segmentation to obtain a spatio-temporal segmentation of objects. Their system computed a ranked list of possible segmentation hypotheses using movement cues. The user could click on the object boundary to carve away the wrong ones iteratively. **[Practical peer prediction for peer assessment](http://aaai.org/ocs/index.php/HCOMP/HCOMP16/paper/view/14041)** _Victor Shnayder, David C. Parkes, Harvard SEAS_ This work provided empirical analysis of over simulated peer prediction mechanisms on a dataset of 3mil. assessments from edX. It also studied incentives and effect of coordination among participants. They highlighted the problem of inefficiency due to frequent disagreement between peers. **[Extending Workers’ Attention Span Through Dummy Events](http://aaai.org/ocs/index.php/HCOMP/HCOMP16/paper/view/14030)** (_Best paper finalist_) _Avshalom Elmalech, David Sarne, Esther David, Chen Hajaj, Harvard University_ This paper tried to improve the attention span of workers by dynamically augmenting the main task with some artificial events and rewarding the worker upon reporting them. **[Quality estimation of workers in collaborative tasks using group testing](http://aaai.org/ocs/index.php/HCOMP/HCOMP16/paper/view/14026)** (_Our paper_) _Prakhar Ojha, Partha Talukdar, IISc_ In this work, we studied the problem of distinguishing workers from idlers, without assuming any prior knowledge of individual skills and considering "groups" as the smallest observable unit for evaluation. We draw upon literature from group-testing which proposes strategies for forming groups and mechanisms to decode individual qualities from group results. Further, we gave bounds over minimum number of groups required to identify quality of subsets of individuals with high confidence. ### Workshops: There were three different flavored [workshops](http://www.humancomputation.com/2016/participate.html#crowdcamp) organized on the last day of HCOMP. CrowdCamp, GroupSight (Image and video analytics) and Mathematical Foundations of Human Computation. I attended the [MathFoundation](http://chienjuho.com/workshops/mathematical-foundations-of-human-computation/) workshop due to my preference for mathematical modeling and curiosity to know whether leading researchers feel that current frameworks are sufficient to capture nuanced human behavior. The workshop had a few short papers and an amazing lineup of invited talks: [Emery Berger, UMass Amherst,](https://emeryberger.com/) spoke about their system Automan that can take a micro-task and decide to post it to crowd workers with calculated redundancy and obtain high accuracy results. The system automates most of the decisions that a task-requester generally makes and give budget guarantees along with performance assurance. [Adam Tauman Kalai, MSR,](http://research.microsoft.com/en-us/um/people/adum/) spoke on meta algorithms for unsupervised learning. [Manuel Blum, CMU,](https://www.cs.cmu.edu/~mblum/) introduced his amazing idea of Human generated passwords. The work took care of real world cosntraints where humans have to come up with their passwords without much delay (i.e they shouldn’t spend much time calculating it) and that it should be secure from adversaries who do not have the secret key. I really liked this interactive session where Prof. Blum demonstrated his “possible” passwords for unknown websites. [Jaime Teevan, MSR,](http://research.microsoft.com/en-us/um/people/teevan/) presented their cool idea of “microproductivity” where large tasks are broken down into a series of smaller microtasks that can each be completed individually. But she rightly highlighted that it is still not clear which kind of tasks can be broken down and which ones cannot. ### Fun moments: * We ([Madhav](https://madhavcsa.github.io/) and I) reached Austin, TX on halloween weekend. The 6th street was cordoned for halloween celebrations and people poured in with creative/scary costumes. We walked the entire celebration-street twice, ate pizza, grabbed some essential groceries and returned back late night. * Opening reception of HCOMP also promoted creative costumes and was a good socializing event to know people before the conference started. * We entered Austin Museum a little before when it was scheduled to close and walked within the permissible open limits. We later walked long streets of Austin, visited Capitol and freedom history park.  * I also attended [EMNLP](http://www.emnlp2016.net/) (top-tier conference for Natural Language Processing), collocated with HCOMP in Austin. I could attend the best-paper sessions, a keynote talk and several short papers. It was a pleasant surprise to see one of my old school friends at the conference who had come to present a poster. * I developed new love for Mexican and Italian food during my stay there. It was great to meet old friends and recollect past-times over dinner. **CMU Visit:** Soon after the conferences, I had the opportunity of visiting CMU to talk about my work and learn a few of their on-going research projects. Many thanks to [Dr. Partha](http://talukdar.net/) and [Dr. Wolfgang](https://www.andrew.cmu.edu/user/gatt/) for making this possible. I presented my work at seminars in CMU and was happy to see audience excited about my research. While clarifying most of their questions, I gained new viewpoint of my own work. Several new avenues for further exploration became evident after these discussions. I also spent some time thinking about fast-inference methods that can capture higher-order-potentials. These methods would be a strict generalization of pairwise-MRFs to represent richer interaction models between nodes in graphs. It was great to have the company of one our previous lab-mates, [Danish](http://danishpruthi.com) (now student at CMU), for his suggestions of food-joints and help in moving around. Having always stayed in warm-tropical regions of India, I was initially scared to witness sub-zero temperatures of Pittsburgh. Luckily, the days of our stay were fairly sunny and nights weren’t too cold. Most leaves had turned yellow with orangish tinge and the sidewalk trees looked especially beautiful this time. Oh!, and I developed a special liking for salad for meals.  ### Acknowledgements This awesome learning experience was made possible with the help of my research supervisor, home department, MSR and several friends who I owe a great deal. Firstly, I would like to sincerely thank my research supervisor [Dr. Partha]() for his constant support and invaluable advice throughout my project. I would like to acknowledge my [MALL lab-mates](http://malllabiisc.github.io/people/) for constructive discussions and their internal reviews. Much appreciated financial support from [GARP-IISc](http://www.iisc.ac.in/) (Govt. of India) and [Microsoft Research India](https://www.microsoft.com/en-us/research/lab/microsoft-research-india/) Travel Grant that made my travel smooth. Lastly, I want to express my earnest gratitude to [Dr. Wolfgang](https://www.andrew.cmu.edu/user/gatt/) for hosting me at CMU. [Prakhar Ojha](https://prakhar4.github.io/) [prakhar.ojha@csa.iisc.ernet.in](mailto:prakhar.ojha@csa.iisc.ernet.in) [MALL lab, IISc](http://malllabiisc.github.io/)