Let’s talk about feedback systems. In the early days of oDesk (read: 2005), we didn’t have a feedback system in place and I remember being on sales calls where almost every potential customer said, “You know, you guys should build a feedback system. Like eBay.” And it’s not surprising; feedback systems are such a fundamental part of making an online marketplace work that it sticks out like a sore thumb if you don’t have one.
Feedback systems in online marketplaces are designed to improve the user experience as a whole by allowing users to establish their own reputation and review the reputation of others. Users submit feedback on other users and relay their positive or negative experience so others can review feedback before entering into a transaction or relationship. In theory, it all works great. But it depends on a few key principles:
1) Identity Verification - what if any user could create multiple profiles? Well, then there’s not enough motivation to maintain a positive reputation. If I receive a negative feedback score, no worries, I’ll just create a new profile. Problem solved.
2) Transparency - what if people could leave feedback and comments, but others couldn’t see that feedback? Well, that wouldn’t do much good. If a user has received some negative feedback, others need to know about it.
3) Accuracy - the feedback that users leave on the system must be accurate. This seems simple, but in reality is quite complex. There are a lot of factors that go into a user’s decision about whether or not to leave positive or negative feedback.
4) Consistency - users should all “grade” the same. If a user is just a particularly tough critic, it might throw things off.
For the vast majority of cases, feedback systems work great and we believe oDesk’s system works well too. However, I do want to talk about some potential issues with #3 - accuracy of feedback and the mechanisms that may negatively affect accuracy. Let’s take a look at eBay first - they’re the founder of the feast when it comes to feedback systems, and they’ve announced some very major changes to their feedback system. From eBay’s site:
What changes are being made to the eBay Feedback system?
Sellers will no longer be able to leave negative or neutral Feedback for buyers. This change will occur in May, 2008.
Why is eBay making these changes?
No negative or neutral Feedback for buyers
The current system prevents buyers from leaving honest Feedback as they fear retaliation from the sellers if they leave a negative. This makes it hard for buyers to distinguish between sellers while making bidding or buying decisions. In addition, when buyers receive negative Feedback, they reduce their activity in the marketplace, which in-turn harms all sellers.
So, why have eBay buyers not been leaving accurate feedback. I believe it stems from personal motivation. Let’s say I’m an eBay buyer and I had a bad experience. I could leave negative feedback, but that might end up hurting my reputation if I get retaliatory negative feedback. So, why leave negative feedback, it only benefits the next user, not me.
Now let’s look at oDesk. I don’t think it’s as simple as the eBay feedback system since eBay is all about point transactions. The transaction is arranged, paid, and completed all within a couple days. At oDesk, it’s all about relationships, which are often ongoing and may result in repeat transactions. So, let’s look at some of the possible issues that may arise near the end of a relationship? Let’s say that buyer x has been working with provider y for 3 months on a website. It’s nearing completion, and now there are a few things that might happen (note: oDesk’s feedback system is bi-directional with a 1-5 scale):
- If everything went great, both parties will know that perfect 5 feedback scores will result.
- If the provider is expecting a 4-4.9 feedback, provider might email buyer explicitly asking for a 5.
- If the provider is expecting a lower score, the provider may, in the worst cases, blackmail the buyer. eg, “I’ll take down your site, if you don’t give me a good feedback score.” If you’re buyer x, you might as well give a good feedback score since you don’t want to risk your site coming down.
So, the provider looks to be malicious in the example above. What about buyers? Are there ever any issues that crop up with buyers of services? Here’s a couple examples that I’ve heard:
- So, it’s been 3 months and the buyer has exceeded their budget but has changing requirements and still needs more work. The buyer may threaten bad feedback scores unless the provider finishes the remaining work for free.
- The buyer is actually quite happy with the work of the provider and needs to retain full-time work. The buyer, being of low moral character, does not want to give the provider a good feedback score. Why? Because the buyer fears losing the provider to other interested buyers. Or because the buyer does not want the provider to increase the hourly rate based on a strong reputation.
While these issues are the minority of cases with online feedback systems, they have a negative effect on the whole system. I believe the only way to ultimately resolve the issues is to foster open, honest discussion about the feedback system. This post is just a problem statement, solutions are ‘to be determined’. It seems that eBay hasn’t figured it out yet, and I can certainly say that oDesk hasn’t figured everything out. But I can say that we’re dedicated to focus on user quality, and there’s no doubt that the feedback system and its integrity is critical to the success of the marketplace.
ps - if you know the solution, please comment. 
- A message from their site: 
So, if you are a provider and looking to build out a robust online profile, consider finding an interesting open-source project and contributing however you can. You will always be able to share this work with anyone and demonstrate your performance. Head on over to 
