Crowdsourcing

The term crowdsourcing was coined in 2006 by two editors at Wired, Jeff Howe and Mark Robinson, to describe how businesses were using the Internet to "outsource work to the crowd", which quickly led to the portmanteau "crowdsourcing". The online dictionary Merriam-Webster defines it as: "the practice of obtaining needed services, ideas, or content by soliciting contributions from a large group of people and especially from the online community rather than from traditional employees or suppliers." Daren C. Brabham defined crowdsourcing as an "online, distributed problem-solving and production model." Despite the multiplicity of definitions for crowdsourcing, one constant has been the broadcasting of problems to the public, and an open call for contributions to help solve the problem. Members of the public submit solutions that are then owned by the entity who originally broadcast the problem. In some cases, the contributor of the solution is compensated monetarily with prizes or public recognition. In other cases, the only rewards may be praise or intellectual satisfaction. Crowdsourcing may produce solutions from amateurs or volunteers working in their spare time, from experts, or from small businesses. ==Historical examples==

While the term "crowdsourcing" was popularized online to describe Internet-based activities, The fragmentation meant that the government had to reform the tax system relying more on the taxation of salt and most importantly business leading to the creation of the Joint-Stock Company. • 1714 – The longitude rewards: When the British government was trying to find a way to measure a ship's longitudinal position, they offered the public a monetary prize to whoever came up with the best solution. • 1783 – King Louis XVI offered an award to the person who could "make the alkali" by decomposing sea salt by the "simplest and most economic method". • 1849 – A network of some 150 volunteer weather observers all over the USA was set up as a part of the Smithsonian Institution's Meteorological Project started by the Smithsonian's first Secretary, Joseph Henry, who used the telegraph to gather volunteers' data and create a large weather map, making new information available to the public daily. For instance, volunteers tracked a tornado passing through Wisconsin and sent the findings via telegraph to the Smithsonian. Henry's project is considered the origin of what later became the National Weather Service. Within a decade, the project had more than 600 volunteer observers and had spread to Canada, Mexico, Latin America, and the Caribbean. • 1884 – Publication of the Oxford English Dictionary: 800 volunteers catalogued words to create the first fascicle of the OED. • 1979 – Robert Axelrod invited academics on-line to submit FORTRAN algorithms to play the repeated Prisoner's Dilemma; A tit for tat algorithm ended up in first place. • 1981 - Jilly Cooper gathered stories about mongrels for the book Intelligent and Loyal, by putting an advert in newspapers asking people to share stories about their pets for the book. • 1983 – Richard Stallman began work on the GNU operating system. Programmers from around the world contribute to the GNU operating system. Linux kernel is one of the kernels used in this operating system, thus forming the GNU/Linux operating system, which many people call as Linux. • 1996 – The Hollywood Stock Exchange was founded: It allowed buying and selling of shares. • 1999– The U.S. Geological Survey's (USGS's) "Did You Feel It?" website was used in the US as a method where by residents could report any tremors or shocks they felt from a recent earthquake and the approximate magnitude of the earthquake. • 2000 – JustGiving was established: This online platform allows the public to help raise money for charities. • 2000 – iStockPhoto was founded: The free stock imagery website allows the public to contribute to and receive commission for their contributions. • 2001 – Launch of Wikipedia: "Free-access, free content Internet encyclopedia". • 2001 – Foundation of Topcoder – crowdsourcing software development company. • 2004 – OpenStreetMap, a collaborative project to create a free editable map of the world, was launched. • 2004 – Toyota's first "Dream car art" contest: Children were asked globally to draw their "dream car of the future". • 2005 – Kodak's "Go for the Gold" contest: Kodak asked anyone to submit a picture of a personal victory. • 2005 – Reddit was launched in 2005. Reddit is a social media platform and online community where users can submit, discuss and vote, leading to diverse discussions and interactions. • 2009 – Waze (then named FreeMap Israel), a community-oriented GPS app, was created. It allows users to submit road information and route data based on location, such as reports of car accidents or traffic, and integrates that data into its routing algorithms for all users of the app. • 2010 - Following the Deepwater Horizon oil spill, BP initiated a crowdsourcing effort called the "Deepwater Horizon Response," inviting external experts and the public to submit innovative ideas and technical solutions for containing and cleaning up the massive oil spill. This initiative aimed to leverage collective intelligence to address the unprecedented environmental disaster. • 2010 – The 1947 Partition Archive, an oral history project that asked community members around the world to document oral histories from aging witnesses of a significant but under-documented historical event, the 1947 Partition of India, was founded. • 2011 – Casting of Flavours (Do us a flavor in the USA) – a campaign launched by PepsiCo's Lay's in Spain. The campaign was to create a new flavor for the snack where the consumers were directly involved in its formation. • 2012 - Open Food Facts, a collaborative project to create a libre encyclopedia of food products in the world using smartphones, is launched, followed by extensions on cosmetics, pet food, other products and prices. • 2016 - The Syngenta Crop Challenge, organized with the Institute for Operations Research and the Management Sciences (INFORMS), tasked participants to develop models for predicting optimal seed varieties for farmers to maximize crop yield. The competition focused on applying advanced analytics to address agricultural challenges, with finalists presenting at the INFORMS Analytics Conference. • 2017 - Syngenta partnered with the AI for Good Foundation to launch the Syngenta AI Challenge, an international competition focused on using artificial intelligence and machine learning for plant breeding. Participants were provided datasets including seed genetic information, soil, weather and climate data to develop models for analyzing seed data more effectively. • 2019 - Principal Financial Group partnered with Carnegie Mellon University's Department of Statistics and Data Science on crowdsourced research projects. The collaboration involved concurrent student projects across three of CMU's colleges, resulting in statistical models combining real estate returns with predictions across diverse markets and property types. Early competitions Crowdsourcing has often been used in the past as a competition to discover a solution. The French government proposed several of these competitions, often rewarded with Montyon Prizes. These included the Leblanc process, or the Alkali prize, where a reward was provided for separating the salt from the alkali, and the Fourneyron's turbine, when the first hydraulic commercial turbine was developed. In response to a challenge from the French government, Nicolas Appert won a prize for inventing a new way of food preservation that involved sealing food in air-tight jars. The British government provided a similar reward to find an easy way to determine a ship's longitude in the Longitude Prize. During the Great Depression, out-of-work clerks tabulated higher mathematical functions in the Mathematical Tables Project as an outreach project. One of the largest crowdsourcing campaigns was a public design contest in 2010 hosted by the Indian government's finance ministry to create a symbol for the Indian rupee. Thousands of people sent in entries before the government zeroed in on the final symbol based on the Devanagari script using the letter Ra. == Applications ==

A number of motivations exist for businesses to use crowdsourcing to accomplish their tasks. These include the ability to offload peak demand, access cheap labor and information, generate better results, access a wider array of talent than what is present in one organization, and undertake problems that would have been too difficult to solve internally. Crowdsourcing allows businesses to submit problems on which contributors can work—on topics such as science, manufacturing, biotech, and medicine—optionally with monetary rewards for successful solutions. Although crowdsourcing complicated tasks can be difficult, simple work tasks can be crowdsourced cheaply and effectively. Crowdsourcing also has the potential to be a problem-solving mechanism for government and nonprofit use. Urban and transit planning are prime areas for crowdsourcing. For example, from 2008 to 2009, a crowdsourcing project for transit planning in Salt Lake City was created to test the public participation process. Another notable application of crowdsourcing for government problem-solving is Peer-to-Patent, which was an initiative to improve patent quality in the United States through gathering public input in a structured, productive manner. Researchers have used crowdsourcing systems such as Amazon Mechanical Turk or CloudResearch to aid their research projects by crowdsourcing some aspects of the research process, such as data collection, parsing, and evaluation to the public. Notable examples include using the crowd to create speech and language databases, to conduct user studies, Crowdsourcing systems provided researchers with the ability to gather large amounts of data, and helped researchers to collect data from populations and demographics they may not have access to locally. Artists have also used crowdsourcing systems. In a project called the Sheep Market, Aaron Koblin used Mechanical Turk to collect 10,000 drawings of sheep from contributors around the world. Artist Sam Brown leveraged the crowd by asking visitors of his website explodingdog to send him sentences to use as inspirations for his paintings. Art curator Andrea Grover argues that individuals tend to be more open in crowdsourced projects because they are not being physically judged or scrutinized. As with other types of uses, artists use crowdsourcing systems to generate and collect data. The crowd also can be used to provide inspiration and to collect financial support for an artist's work. In navigation systems, crowdsourcing from 100 million drivers were used by INRIX to collect users' driving times to provide better GPS routing and real-time traffic updates. In recent years, crowdsourcing has also been widely used in artificial intelligence training and data labeling. In healthcare The use of crowdsourcing in medical and health research is increasing systematically. The process involves outsourcing tasks or gathering input from a large, diverse groups of people, often facilitated through digital platforms, to contribute to medical research, diagnostics, data analysis, promotion, and various healthcare-related initiatives. Usage of this innovative approach supplies a useful community-based method to improve medical services. From funding individual medical cases and innovative devices to supporting research, community health initiatives, and crisis responses, crowdsourcing proves its versatile impact in addressing diverse healthcare challenges. In 2011, UNAIDS initiated the participatory online policy project to better engage young people in decision-making processes related to AIDS. The project acquired data from 3,497 participants across seventy-nine countries through online and offline forums. The outcomes generally emphasized the importance of youth perspectives in shaping strategies to effectively address AIDS which provided a valuable insight for future community empowerment initiatives. Another approach is sourcing results of clinical algorithms from collective input of participants. Researchers from SPIE developed a crowdsourcing tool, to train individuals, especially middle and high school students in South Korea, to diagnose malaria-infected red blood cells. Using a statistical framework, the platform combined expert diagnoses with those from minimally trained individuals, creating a gold standard library. The objective was to swiftly teach people to achieve great diagnosis accuracy without any prior training. Cancer medicine journal conducted a review of the studies published between January 2005 and June 2016 on crowdsourcing in cancer research, with the usage PubMed, CINAHL, Scopus, PsychINFO, and Embase. All of them strongly advocate for continuous efforts to refine and expand crowdsourcing applications in academic scholarship. Analysis highlighted the importance of interdisciplinary collaborations and widespread dissemination of knowledge; the review underscored the need to fully harness crowdsourcing's potential to address challenges within cancer research. These responses helped him to make a series of scientific breakthroughs including observing the fact that meteor showers are seen nationwide and fall from space under the influence of gravity. The responses also allowed him to approximate a velocity for the meteors. More recent versions of crowdsourcing in astronomy are NASA's photo organizing project, which asked internet users to browse photos taken from space and try to identify the location the picture is documenting, and the 2007 Galaxy Zoo project. Behavioral science In the field of behavioral science, crowdsourcing is often used to gather data and insights on human behavior and decision making. Researchers may create online surveys or experiments that are completed by a large number of participants, allowing them to collect a diverse and potentially large amount of data. The use of crowdsourcing in behavioral science has the potential to greatly increase the scope and efficiency of research, and has been used in studies on topics such as psychology research, political attitudes, and social media use. Energy system research Energy system models require large and diverse datasets, increasingly so given the trend towards greater temporal and spatial resolution. In response, there have been several initiatives to crowdsource this data. Launched in December 2009, OpenEI is a collaborative website run by the US government that provides open energy data. While much of its information is from US government sources, the platform also seeks crowdsourced input from around the world. The semantic wiki and database Enipedia also publishes energy systems data using the concept of crowdsourced open information. Enipedia went live in March 2011. Genealogy research Genealogical research used crowdsourcing techniques long before personal computers were common. Beginning in 1942, members of the Church of Jesus Christ of Latter-day Saints encouraged members to submit information about their ancestors. The submitted information was gathered together into a single collection. In 1969, to encourage more participation, the church started the three-generation program. In this program, church members were asked to prepare documented family group record forms for the first three generations. The program was later expanded to encourage members to research at least four generations and became known as the four-generation program. Institutes that have records of interest to genealogical research have used crowds of volunteers to create catalogs and indices to records. Genetic genealogy research Genetic genealogy is a combination of traditional genealogy with genetics. The rise of personal DNA testing, after the turn of the century, by companies such as Gene by Gene, FTDNA, GeneTree, 23andMe, and Ancestry.com, has led to public and semi public databases of DNA testing using crowdsourcing techniques. Citizen science projects have included support, organization, and dissemination of personal DNA (genetic) testing. Similar to amateur astronomy, citizen scientists encouraged by volunteer organizations like the International Society of Genetic Genealogy have provided valuable information and research to the professional scientific community. The Genographic Project, which began in 2005, is a research project carried out by the National Geographic Society's scientific team to reveal patterns of human migration using crowdsourced DNA testing and reporting of results. Ornithology Another early example of crowdsourcing occurred in the field of ornithology. On 25 December 1900, Frank Chapman, an early officer of the National Audubon Society, initiated a tradition dubbed the "Christmas Day Bird Census". The project called birders from across North America to count and record the number of birds in each species they witnessed on Christmas Day. The project was successful, and the records from 27 different contributors were compiled into one bird census, which tallied around 90 species of birds. This large-scale collection of data constituted an early form of citizen science, the premise upon which crowdsourcing is based. In the 2012 census, more than 70,000 individuals participated across 2,369 bird count circles. Christmas 2014 marked the National Audubon Society's 115th annual Christmas Bird Count. ==== Seismology ==== The European-Mediterranean Seismological Centre (EMSC) has developed a seismic detection system by monitoring the traffic peaks on its website and analyzing keywords used on Twitter. In journalism Crowdsourcing is increasingly used in professional journalism. Journalists are able to organize crowdsourced information by fact checking the information, and then using the information they have gathered in their articles as they see fit. A daily newspaper in Sweden has successfully used crowdsourcing in investigating the home loan interest rates in the country in 2013–2014, which resulted in over 50,000 submissions. A daily newspaper in Finland crowdsourced an investigation into stock short-selling in 2011–2012, and the crowdsourced information led to revelations of a tax evasion system by a Finnish bank. The bank executive was fired and policy changes followed. TalkingPointsMemo in the United States asked its readers to examine 3,000 emails concerning the firing of federal prosecutors in 2008. The British newspaper The Guardian crowdsourced the examination of hundreds of thousands of documents in 2009. Data donation Data donation is a crowdsourcing approach to gather digital data. It is used by researchers and organizations to gain access to data from online platforms, websites, search engines and apps and devices. Data donation projects usually rely on participants volunteering their authentic digital profile information. Examples include: • DataSkop developed by Algorithm Watch, a non-profit research organization in Germany, which accessed data on social media algorithms and automated decision-making systems. • Mozilla Rally, from the Mozilla Foundation, is a browser extension for adult participants in the US to provide access to their data for research projects. • The Australian Search Experience and Ad Observatory projects set up in 2021 by researchers at the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S) in Australia was using data donations to analyze how Google personalized search results, and examine how Facebook's algorithmic advertising model worked. • The Citizen Browser Project, developed by The Markup, was designed to measure how disinformation traveled across social media platforms over time. • Large Emergency Event Digital Information Repository was an effort to create a repository for images and videos from natural disasters, terrorist, and criminal events In social media Crowdsourcing is used in large scale media, such as the community notes system of the X platform. Crowdsourcing on such platforms is thought to be effective in combating partisan misinformation on social media when certain conditions are met. Success may depend on trust in fact-checking sources, the ability to present information that challenges previous beliefs without causing excessive dissonance, and having a sufficiently large and diverse crowd of participants. Effective crowdsourcing interventions must navigate politically polarized environments where trusted sources may be less inclined to provide dissonant opinions. By leveraging network analysis to connect users with neighboring communities outside their ideological echo chambers, crowdsourcing can provide an additional layer of content moderation. In public policy Crowdsourcing public policy and the production of public services is also referred to as citizen sourcing. While some scholars argue crowdsourcing for this purpose as a policy tool or a definite means of co-production, others question that and argue that crowdsourcing should be considered just as a technological enabler that simply increases speed and ease of participation. Crowdsourcing can also play a role in democratization. The first conference focusing on Crowdsourcing for Politics and Policy took place at Oxford University, under the auspices of the Oxford Internet Institute in 2014. Research has emerged since 2012 which focused on the use of crowdsourcing for policy purposes. These include experimentally investigating the use of Virtual Labor Markets for policy assessment, and assessing the potential for citizen involvement in process innovation for public administration. Governments across the world are increasingly using crowdsourcing for knowledge discovery and civic engagement. Iceland crowdsourced their constitution reform process in 2011, and Finland has crowdsourced several law reform processes to address their off-road traffic laws. The Finnish government allowed citizens to go on an online forum to discuss problems and possible resolutions regarding some off-road traffic laws. The crowdsourced information and resolutions would then be passed on to legislators to refer to when making a decision, allowing citizens to contribute to public policy in a more direct manner. Palo Alto crowdsources feedback for its Comprehensive City Plan update in a process started in 2015. The House of Representatives in Brazil has used crowdsourcing in policy-reforms. For part of the Obama and Trump Administrations, the We the People system collected signatures on petitions, which were entitled to an official response from the White House once a certain number had been reached. Several U.S. federal agencies ran inducement prize contests, including NASA and the Environmental Protection Agency. Crowdsourcing language-related data online has proven very effective and many dictionary compilation projects used crowdsourcing. It is used particularly for specialist topics and languages that are not well documented, such as for the Oromo language. Software programs have been developed for crowdsourced dictionaries, such as WeSay. A slightly different form of crowdsourcing for language data was the online creation of scientific and mathematical terminology for American Sign Language. In linguistics, crowdsourcing strategies have been applied to estimate word knowledge, vocabulary size, and word origin. Implicit crowdsourcing on social media has also approximating sociolinguistic data efficiently. Reddit conversations in various location-based subreddits were analyzed for the presence of grammatical forms unique to a regional dialect. These were then used to map the extent of the speaker population. The results could roughly approximate large-scale surveys on the subject without engaging in field interviews. Mining publicly available social media conversations can be used as a form of implicit crowdsourcing to approximate the geographic extent of speaker dialects. Crowdsourcing has been extensively used to collect high-quality gold standards for creating automatic systems in natural language processing (e.g. named entity recognition, entity linking). In product design Organizations often leverage crowdsourcing to gather ideas for new products as well as for the refinement of established products. Lego allows users to work on new product designs while conducting requirements testing. Any user can provide a design for a product, and other users can vote on the product. Once the submitted product has received 10,000 votes, it will be formally reviewed in stages and go into production with no impediments such as legal flaws identified. The creator receives royalties from the net income. Labelling new products as "customer-ideated" through crowdsourcing initiatives, as opposed to not specifying the source of design, leads to a substantial increase in the actual market performance of the products. Merely highlighting the source of design to customers, particularly, attributing the product to crowdsourcing efforts from user communities, can lead to a significant boost in product sales. Consumers perceive "customer-ideated" products as more effective in addressing their needs, leading to a quality inference. The design mode associated with crowdsourced ideas is considered superior in generating promising new products, contributing to the observed increase in market performance. In business Crowdsourcing is widely used by businesses to source feedback and suggestions on how to improve their products and services. They have to pay a booking fee every time they book a room. The landlord, in turn, pays a service fee for the amount due. The company has 1,500 properties in 34,000 cities in more than 190 countries. In market research Crowdsourcing is frequently used in market research as a way to gather insights and opinions from a large number of consumers. Companies may create online surveys or focus groups that are open to the general public, allowing them to gather a diverse range of perspectives on their products or services. This can be especially useful for companies seeking to understand the needs and preferences of a particular market segment or to gather feedback on the effectiveness of their marketing efforts. The use of crowdsourcing in market research allows companies to quickly and efficiently gather a large amount of data and insights that can inform their business decisions. In plant breeding Biotech and life sciences research and development companies have applied crowdsourcing to address complex optimization problems in plant breeding and seed selection. These applications have focused on developing predictive models that analyze genetic, environmental, and agronomic data to improve crop yields under varying conditions. Crowdsourcing approaches in agriculture have included open innovation competitions where participants develop algorithms to predict optimal seed varieties based on factors such as soil characteristics, weather patterns, and climate data. The use of capstone programs in asset management has been characterized by Tom Davenport in a Forbes article as an effective method for organizations to access analytical talent and cognitive diversity while providing students with practical problem-solving experience. Other examples • Geography — Volunteered geographic information (VGI) is geographic information generated through crowdsourcing, as opposed to traditional methods of Professional Geographic Information (PGI). In describing the built environment, VGI has many advantages over PGI, primarily perceived currency, accuracy and authority. OpenStreetMap is an example of crowdsourced mapping project. from the fields and also to provide assistance in removing the weeds. • Libraries, museums and archives — Newspaper text correction at the National Library of Australia was an early, influential example of work with text transcriptions for crowdsourcing in cultural heritage institutions. The Steve Museum project provided a prototype for categorizing artworks. Crowdsourcing is used in libraries for OCR corrections on digitized texts, for tagging and for funding, especially in the absence of financial and human means. Volunteers can contribute explicitly with conscious effort or implicitly without being known by turning the text on the raw newspaper image into human corrected digital form. • Cheating in bridge — Boye Brogeland initiated a crowdsourcing investigation of cheating by top-level bridge players that showed several players as guilty, which led to their suspension. • Open-source software and Crowdsourcing software development have been used extensively in the domain of software development. • Healthcare — Research has emerged that outlined the use of crowdsourcing techniques in the public health domain. The collective intelligence outcomes from crowdsourcing are being generated in three broad categories of public health care: health promotion, and health maintenance. Crowdsourcing also enables researchers to move from small homogeneous groups of participants to large heterogenous groups beyond convenience samples such as students or higher educated people. The SESH group focuses on using crowdsourcing to improve health. ==Methods==

Internet and digital technologies have massively expanded the opportunities for crowdsourcing. However, the effect of user communication and platform presentation can have a major bearing on the success of an online crowdsourcing project. Crowdsourcing can either take an explicit or an implicit route: • Explicit crowdsourcing lets users work together to evaluate, share, and build different specific tasks, while implicit crowdsourcing means that users solve a problem as a side effect of something else they are doing. With explicit crowdsourcing, users can evaluate particular items like books or webpages, or share by posting products or items. Users can also build artifacts by providing information and editing other people's work. • Implicit crowdsourcing can take two forms: standalone and piggyback. Standalone allows people to solve problems as a side effect of the task they are actually doing, whereas piggyback takes users' information from a third-party website to gather information. This is also known as data donation. In his 2013 book, Crowdsourcing, Daren C. Brabham puts forth a problem-based typology of crowdsourcing approaches: • Knowledge discovery and management is used for information management problems where an organization mobilizes a crowd to find and assemble information. It is ideal for creating collective resources. • Distributed human intelligence tasking (HIT) is used for information management problems where an organization has a set of information in hand and mobilizes a crowd to process or analyze the information. It is ideal for processing large data sets that computers cannot easily do. Amazon Mechanical Turk uses this approach. • Broadcast search is used for ideation problems where an organization mobilizes a crowd to come up with a solution to a problem that has an objective, provable right answer. It is ideal for scientific problem-solving. • Peer-vetted creative production is used for ideation problems, where an organization mobilizes a crowd to come up with a solution to a problem which has an answer that is subjective or dependent on public support. It is ideal for design, aesthetic, or policy problems. Ivo Blohm identifies four types of Crowdsourcing Platforms: Microtasking, Information Pooling, Broadcast Search, and Open Collaboration. They differ in the diversity and aggregation of contributions that are created. The diversity of information collected can either be homogenous or heterogenous. The aggregation of information can either be selective or integrative. Some common categories of crowdsourcing have been used effectively in the commercial world include crowdvoting, crowdsolving, crowdfunding, microwork, creative crowdsourcing, crowdsource workforce management, and inducement prize contests. In their conceptual review of the crowdsourcing, Linus Dahlander, Lars Bo Jeppesen, and Henning Piezunka distinguish four steps in the crowdsourcing process: Define, Broadcast, Attract, and Select. Crowdvoting Crowdvoting occurs when a website gathers a large group's opinions and judgments on a certain topic. Some crowdsourcing tools and platforms allow participants to rank each other's contributions, e.g. in answer to the question "What is one thing we can do to make Acme a great company?" One common method for ranking is "like" counting, where the contribution with the most "like" votes ranks first. This method is simple and easy to understand, but it privileges early contributions, which have more time to accumulate votes. In recent years, several crowdsourcing companies have begun to use pairwise comparisons backed by ranking algorithms. Ranking algorithms do not penalize late contributions. They also produce results quicker. Ranking algorithms have proven to be at least 10 times faster than manual stack ranking. One drawback, however, is that ranking algorithms are more difficult to understand than vote counting. The Iowa Electronic Market is a prediction market that gathers crowds' views on politics and tries to ensure accuracy by having participants pay money to buy and sell contracts based on political outcomes. Some of the most famous examples have made use of social media channels: Domino's Pizza, Coca-Cola, Heineken, and Sam Adams have crowdsourced a new pizza, bottle design, beer, and song respectively. A website called Threadless selected the T-shirts it sold by having users provide designs and vote on the ones they like, which are then printed and available for purchase. The California Report Card (CRC), a program jointly launched in January 2014 by the Center for Information Technology Research in the Interest of Society and Lt. Governor Gavin Newsom, is an example of modern-day crowd voting. Participants access the CRC online and vote on six timely issues. Through principal component analysis, the users are then placed into an online "café" in which they can present their own political opinions and grade the suggestions of other participants. This system aims to effectively involve the greater public in relevant political discussions and highlight the specific topics with which people are most concerned. Crowdvoting's value in the movie industry was shown when in 2009 a crowd accurately predicted the success or failure of a movie based on its trailer, a feat that was replicated in 2013 by Google. On Reddit, users collectively rate web content, discussions and comments as well as questions posed to persons of interest in "AMA" and AskScience online interviews. In 2017, Project Fanchise purchased a team in the Indoor Football League and created the Salt Lake Screaming Eagles, a fan run team. Using a mobile app, the fans voted on the day-to-day operations of the team, the mascot name, signing of players and even offensive play calling during games. Crowdfunding Crowdfunding is the process of funding projects by a multitude of people contributing a small amount to attain a certain monetary goal, typically via the Internet. Crowdfunding has been used for both commercial and charitable purposes. The crowdfuding model that has been around the longest is rewards-based crowdfunding. This model is where people can prepurchase products, buy experiences, or simply donate. While this funding may in some cases go towards helping a business, funders are not allowed to invest and become shareholders via rewards-based crowdfunding. Individuals, businesses, and entrepreneurs can showcase their businesses and projects by creating a profile, which typically includes a short video introducing their project, a list of rewards per donation, and illustrations through images. Funders make monetary contribution for numerous reasons: • They connect to the greater purpose of the campaign, such as being a part of an entrepreneurial community and supporting an innovative idea or product. • They connect to a physical aspect of the campaign like rewards and gains from investment. Another example is the Netflix Prize in 2009. People were asked to come up with a recommendation algorithm that is more accurate than Netflix's current algorithm. It had a grand prize of US$1,000,000, and it was given to a team which designed an algorithm that beat Netflix's own algorithm for predicting ratings by 10.06%. Another example of competition-based crowdsourcing is the 2009 DARPA balloon experiment, where DARPA placed 10 balloon markers across the United States and challenged teams to compete to be the first to report the location of all the balloons. A collaboration of efforts was required to complete the challenge quickly and in addition to the competitive motivation of the contest as a whole, the winning team (MIT, in less than nine hours) established its own "collaborapetitive" environment to generate participation in their team. A similar challenge was the Tag Challenge, funded by the US State Department, which required locating and photographing individuals in five cities in the US and Europe within 12 hours based only on a single photograph. The winning team managed to locate three suspects by mobilizing volunteers worldwide using a similar incentive scheme to the one used in the balloon challenge. Using open innovation platforms is an effective way to crowdsource people's thoughts and ideas for research and development. The company InnoCentive is a crowdsourcing platform for corporate research and development where difficult scientific problems are posted for crowds of solvers to discover the answer and win a cash prize that ranges from $10,000 to $100,000 per challenge. Implicit crowdsourcing Implicit crowdsourcing is less obvious because users do not necessarily know they are contributing, yet can still be very effective in completing certain tasks. Rather than users actively participating in solving a problem or providing information, implicit crowdsourcing involves users doing another task entirely where a third party gains information for another topic based on the user's actions. University capstone programs University capstone programs represent a form of crowdsourcing in which organizations sponsor projects completed by student teams as part of academic requirements in analytics, engineering, or business programs. In these programs, companies present problems to students who develop solutions under faculty supervision, often receiving the work at no cost or reduced cost compared to hiring professional consultants. The sponsoring organization gains access to diverse analytical approaches while students gain practical problem-solving experience. Other types • Creative crowdsourcing involves sourcing people for creative projects such as graphic design, crowdsourcing architecture, product design, writing, company naming, illustration, etc. While crowdsourcing competitions have been used for decades in some creative fields such as architecture, creative crowdsourcing has proliferated with the recent development of web-based platforms where clients can solicit a wide variety of creative work at lower cost than by traditional means. • Crowdshipping (crowd-shipping) is a peer-to-peer shipping service, usually conducted via an online platform or marketplace. There are several methods that have been categorized as crowd-shipping: • Travelers heading in the direction of the buyer, and are willing to bring the package as part of their luggage for a reward. • Truck drivers whose route lies along the buyer's location and who are willing to take extra items in their truck. • Community-based platforms that connect international buyers and local forwarders, by allowing buyers to use forwarder's address as purchase destination, after which forwarders ship items further to the buyer. • Crowdsolving is a collaborative and holistic way of solving a problem through many people, communities, groups, or resources. It is a type of crowdsourcing with focus on complex and intellectually demanding problems requiring considerable effort, and the quality or uniqueness of contribution. • Problem–idea chains are a form of idea crowdsourcing and crowdsolving, where individuals are asked to submit ideas to solve problems and then problems that can be solved with those ideas. The aim is to find encourage individuals to find practical solutions to problems that are well thought through. • Macrowork tasks typically have these characteristics: they can be done independently, they take a fixed amount of time, and they require special skills. Macro-tasks could be part of specialized projects or could be part of a large, visible project where workers pitch in wherever they have the required skills. The key distinguishing factors are that macro-work requires specialized skills and typically takes longer, while microwork requires no specialized skills. • Microwork is a crowdsourcing platform that allows users to do small tasks for which computers lack aptitude in for low amounts of money. Amazon's Mechanical Turk has created many different projects for users to participate in, where each task requires very little time and offers a very small amount in payment. An example of a Mechanical Turk project is when users searched satellite images for a boat to find Jim Gray, a missing computer scientist. This allows for real-time data gathering and gives projects greater reach and accessibility. However, mobile crowdsourcing can lead to an urban bias, and can have safety and privacy concerns. • Simple projects are those that require a large amount of time and skills compared to micro and macro-work. While an example of macro-work would be writing survey feedback, simple projects rather include activities like writing a basic line of code or programming a database, which both require a larger time commitment and skill level. These projects are usually not found on sites like Amazon Mechanical Turk, and are rather posted on platforms like Upwork that call for a specific expertise. • Complex projects generally take the most time, have higher stakes, and call for people with very specific skills. These are generally "one-off" projects that are difficult to accomplish and can include projects such as designing a new product that a company hopes to patent. Such projects are considered to be complex because design is a meticulous process that requires a large amount of time to perfect, and people completing the project must have specialized training in design to effectively complete the project. These projects usually pay the highest, yet are rarely offered. • Crowdsourcing-Based Optimization refers to a class of methods that utilize crowdsourcing to enable a group of workers to collaboratively collect data and solve optimization problems related to the data. Due to the heterogeneity of workers, the data collected varies, and the workers' understanding of the optimization problems also differs, thus posing challenges to collaboratively solve a global optimization problem. Representative methods for solving Crowdsourcing-Based Optimization include CrowdEC, which is a mechanism that dispatch the optimization tasks to a group of workers that collaborate to perform evolutionary computation (EC) in a distributed manner. ==Demographics of the crowd==

The crowd is an umbrella term for the people who contribute to crowdsourcing efforts. Though it is sometimes difficult to gather data about the demographics of the crowd as a whole, several studies have examined various specific online platforms. Amazon Mechanical Turk has received a great deal of attention in particular. A study in 2008 by Ipeirotis found that users at that time were primarily American, young, female, and well-educated, with 40% earning more than $40,000 per year. In November 2009, Ross found a very different Mechanical Turk population where 36% of which was Indian. Two-thirds of Indian workers were male, and 66% had at least a bachelor's degree. Two-thirds had annual incomes less than $10,000, with 27% sometimes or always depending on income from Mechanical Turk to make ends meet. More recent studies have found that U.S. Mechanical Turk workers are approximately 58% female, and nearly 67% of workers are in their 20s and 30s. Close to 80% are White, and 9% are Black. MTurk workers are less likely to be married or have children as compared to the general population. In the US population over 18, 45% are unmarried, while the proportion of unmarried workers on MTurk is around 57%. Additionally, about 55% of MTurk workers do not have any children, which is significantly higher than the general population. Approximately 68% of U.S. workers are employed, compared to 60% in the general population. MTurk workers in the U.S. are also more likely to have a four-year college degree (35%) compared to the general population (27%). Politics within the U.S. sample of MTurk are skewed liberal, with 46% Democrats, 28% Republicans, and 26% "other". MTurk workers are also less religious than the U.S. population, with 41% religious, 20% spiritual, 21% agnostic, and 16% atheist. The demographics of Microworkers.com differ from Mechanical Turk in that the US and India together accounting for only 25% of workers; 197 countries are represented among users, with Indonesia (18%) and Bangladesh (17%) contributing the largest share. However, 28% of employers are from the US. Another study of the demographics of the crowd at iStockphoto found a crowd that was largely white, middle- to upper-class, higher educated, worked in a so-called "white-collar job" and had a high-speed Internet connection at home. In a crowdsourcing diary study of 30 days in Europe, the participants were predominantly higher educated women. Motivations Contributors Many researchers suggest that both intrinsic and extrinsic motivations cause people to contribute to crowdsourced tasks and these factors influence different types of contributors. For example, people employed in a full-time position rate human capital advancement as less important than part-time workers do, while women rate social contact as more important than men do. such as the altruistic motivations of online volunteers. Chandler and Kapelner found that US users of the Amazon Mechanical Turk were more likely to complete a task when told they were going to help researchers identify tumor cells, than when they were not told the purpose of their task. However, of those who completed the task, quality of output did not depend on the framing. Motivation in crowdsourcing is often a mix of intrinsic and extrinsic factors. In a crowdsourced law-making project, the crowd was motivated by both intrinsic and extrinsic factors. Intrinsic motivations included fulfilling civic duty, affecting the law for sociotropic reasons, to deliberate with and learn from peers. Extrinsic motivations included changing the law for financial gain or other benefits. Participation in crowdsourced policy-making was an act of grassroots advocacy, whether to pursue one's own interest or more altruistic goals, such as protecting nature. Another form of social motivation is prestige or status. The International Children's Digital Library recruited volunteers to translate and review books. Because all translators receive public acknowledgment for their contributions, Kaufman and Schulz cite this as a reputation-based strategy to motivate individuals who want to be associated with institutions that have prestige. The Mechanical Turk uses reputation as a motivator in a different sense, as a form of quality control. Crowdworkers who frequently complete tasks in ways judged to be inadequate can be denied access to future tasks, whereas workers who pay close attention may be rewarded by gaining access to higher-paying tasks or being on an "Approved List" of workers. This system may incentivize higher-quality work. However, this system only works when requesters reject bad work, which many do not. Despite the potential global reach of IT applications online, recent research illustrates that differences in location affect participation outcomes in IT-mediated crowds. ==Limitations and controversies==

While there it lots of anecdotal evidence that illustrates the potential of crowdsourcing and the benefits that organizations have derived, there is scientific evidence that crowdsourcing initiatives often fail. At least six major topics cover the limitations and controversies about crowdsourcing: • Failure to attract contributions • Impact of crowdsourcing on product quality • Entrepreneurs contribute less capital themselves • Increased number of funded ideas • The value and impact of the work received from the crowd • The ethical implications of low wages paid to workers • Trustworthiness and informed decision making Failure to attract contributions Crowdsourcing initiatives often fail to attract sufficient or beneficial contributions. The vast majority of crowdsourcing initiatives hardly attract contributions; an analysis of thousands of organizations' crowdsourcing initiatives illustrates that only the 90th percentile of initiatives attracts more than one contribution a month. Companies, or additional crowdworkers, then have to sort through the low-quality contributions. The task of sorting through crowdworkers' contributions, along with the necessary job of managing the crowd, requires companies to hire actual employees, thereby increasing management overhead. For example, susceptibility to faulty results can be caused by targeted, malicious work efforts. Since crowdworkers completing microtasks are paid per task, a financial incentive often causes workers to complete tasks quickly rather than well. Some companies, like CloudResearch, control data quality by repeatedly vetting crowdworkers to ensure they are paying attention and providing high-quality work. argue that, the prevailing practice of modeling crowdsourcing data collection tasks in terms of fixed classes (options), unnecessarily restricts quality. Results demonstrate that information accuracy depends on the classes used to model domains, with participants providing more accurate information when classifying phenomena at a more general level (which is typically less useful to sponsor organizations, hence less common). Further, greater overall accuracy is expected when participants could provide free-form data compared to tasks in which they select from constrained choices. In behavioral science research, it is often recommended to include open-ended responses, in addition to other forms of attention checks, to assess data quality. Just as limiting, oftentimes there is not enough skills or expertise in the crowd to successfully accomplish the desired task. While this scenario does not affect "simple" tasks such as image labeling, it is particularly problematic for more complex tasks, such as engineering design or product validation. A comparison between the evaluation of business models from experts and an anonymous online crowd showed that an anonymous online crowd cannot evaluate business models to the same level as experts. In these cases, it may be difficult or even impossible to find qualified people in the crowd, as their responses represent only a small fraction of the workers compared to consistent, but incorrect crowd members. However, if the task is "intermediate" in its difficulty, estimating crowdworkers' skills and intentions and leveraging them for inferring true responses works well, albeit with an additional computation cost. Crowdworkers are a nonrandom sample of the population. Many researchers use crowdsourcing to quickly and cheaply conduct studies with larger sample sizes than would be otherwise achievable. However, due to limited access to the Internet, participation in low developed countries is relatively low. Participation in highly developed countries is similarly low, largely because the low amount of pay is not a strong motivation for most users in these countries. These factors lead to a bias in the population pool towards users in medium developed countries, as deemed by the human development index. Participants in these countries sometimes masquerade as U.S. participants to gain access to certain tasks. This led to the "bot scare" on Amazon Mechanical Turk in 2018, when researchers thought bots were completing research surveys due to the lower quality of responses originating from medium-developed countries. The likelihood that a crowdsourced project will fail due to lack of monetary motivation or too few participants increases over the course of the project. Tasks that are not completed quickly may be forgotten, buried by filters and search procedures. This results in a long-tail power law distribution of completion times. Additionally, low-paying research studies online have higher rates of attrition, with participants not completing the study once started. One of the problems of crowdsourcing products is the lack of interaction between the crowd and the client. Usually little information is known about the final product, and workers rarely interacts with the final client in the process. This can decrease the quality of product as client interaction is considered to be a vital part of the design process. An additional cause of the decrease in product quality that can result from crowdsourcing is the lack of collaboration tools. In a typical workplace, coworkers are organized in such a way that they can work together and build upon each other's knowledge and ideas. Furthermore, the company often provides employees with the necessary information, procedures, and tools to fulfill their responsibilities. However, in crowdsourcing, crowd-workers are left to depend on their own knowledge and means to complete tasks. Entrepreneurs contribute less capital themselves To make an idea turn into a reality, the first component needed is capital. Depending on the scope and complexity of the crowdsourced project, the amount of necessary capital can range from a few thousand dollars to hundreds of thousands, if not more. The capital-raising process can take from days to months depending on different variables, including the entrepreneur's network and the amount of initial self-generated capital. The crowdsourcing process allows entrepreneurs to access a wide range of investors who can take different stakes in the project. As an effect, crowdsourcing simplifies the capital-raising process and allows entrepreneurs to spend more time on the project itself and reaching milestones rather than dedicating time to get it started. Overall, the simplified access to capital can save time to start projects and potentially increase the efficiency of projects. Others argue that easier access to capital through a large number of smaller investors can hurt the project and its creators. With a simplified capital-raising process involving more investors with smaller stakes, investors are more risk-seeking because they can take on an investment size with which they are comfortable. Increased number of funded ideas The raw number of ideas that get funded and the quality of the ideas is a large controversy over the issue of crowdsourcing. Proponents argue that crowdsourcing is beneficial because it allows the formation of startups with niche ideas that would not survive venture capitalist or angel funding, which are oftentimes the primary investors in startups. Many ideas are scrapped in their infancy due to insufficient support and lack of capital, but crowdsourcing allows these ideas to be started if an entrepreneur can find a community to take interest in the project. Crowdsourcing allows those who would benefit from the project to fund and become a part of it, which is one way for small niche ideas get started. However, when the number of projects grows, the number of failures also increases. Crowdsourcing assists the development of niche and high-risk projects due to a perceived need from a select few who seek the product. With high risk and small target markets, the pool of crowdsourced projects faces a greater possible loss of capital, lower return, and lower levels of success. Labor-related concerns Because crowdworkers are considered independent contractors rather than employees, they are not guaranteed minimum wage. In practice, workers using Amazon Mechanical Turk generally earn less than minimum wage. In 2009, it was reported that United States Turk users earned an average of $2.30 per hour for tasks, while users in India earned an average of $1.58 per hour, which is below minimum wage in the United States (but not in India). In 2018, a survey of 2,676 Amazon Mechanical Turk workers doing 3.8 million tasks found that the median hourly wage was approximately $2 per hour, and only 4% of workers earned more than the federal minimum wage of $7.25 per hour. Some researchers who have considered using Mechanical Turk to get participants for research studies have argued that the wage conditions might be unethical. However, according to other research, workers on Amazon Mechanical Turk do not feel they are exploited and are ready to participate in crowdsourcing activities in the future. A more recent study using stratified random sampling to access a representative sample of Mechanical Turk workers found that the U.S. MTurk population is financially similar to the general population. Critics say that crowdsourcing arrangements exploit individuals in the crowd, and a call has been made for crowds to organize for their labor rights. Collaboration between crowd members can also be difficult or even discouraged, especially in the context of competitive crowd sourcing. Crowdsourcing site InnoCentive allows organizations to solicit solutions to scientific and technological problems; only 10.6% of respondents reported working in a team on their submission. Amazon Mechanical Turk workers collaborated with academics to create a platform, WeAreDynamo.org, that allows them to organize and create campaigns to better their work situation, but the site is no longer running. Another platform run by Amazon Mechanical Turk workers and academics, Turkopticon, continues to operate and provides worker reviews on Amazon Mechanical Turk employers. America Online settled the case Hallissey et al. v. America Online, Inc. for $15 million in 2009, after unpaid moderators sued to be paid the minimum wage as employees under the U.S. Fair Labor Standards Act. Crowdsourcing has also been increasingly used in artificial intelligence training, where large datasets of human-labeled images or text are collected to improve machine learning models. Other concerns Besides insufficient compensation and other labor-related disputes, there have also been concerns regarding privacy violations, the hiring of vulnerable groups, breaches of anonymity, psychological damage, the encouragement of addictive behaviors, and more. Many but not all of the issues related to crowdworkes overlap with concerns related to content moderators. == See also ==