Mods Rating System on WoWI

[Cairenn]

Yesterday after an extensive staff discussion, WoWInterface removed the rating system from the addons portion of the site. It has come to our attention that over the course of the past few years, people have tried to "game" the ratings system by having their friends visit the site to rate up a mod, or to launch a "rate down" attack against a particular addon or author. Clearly this sort of behavior is unacceptable and of no benefit to the addon community. In addition authors and users alike have reported that the concept of a subjective rating on addons is confusing in the long run.

We would like to hear your suggestions about a rating system (or something similar) that would be more beneficial to the users and authors of WoWInterface.com. Obviously the favorites system still exists, but as always that is meant for convenience and not as a competition. Some questions to start the discussion:

1. Does a ratings system help the community promote new and exciting mods, or does it just turn everyone into BorgUI 4.3?
2. Would a positive system that only allowed users to flag a mod upwards be beneficial, or would it require too much administration/policing to prevent someone from launching a "ZOMG PROMOTE MY MOD111!1" campaign?
3. How else can we work to spread the news about the exciting user interfaces that are available on WoWInterface other than the current methods which show the top favorites, new addons, top downloads, our addon spotlight on the addons tab and our weekly picks, etc.?

[erica647]

1. Does a ratings system help the community promote new and exciting mods, or does it just turn everyone into BorgUI 4.3?

The ratings system can be beneficial imo if some sort of requirement would be in place requiring the user to at least give some reason as to why they gave a particular mod a low rating. Maybe something like a required field that has to be filled in for a certain level of rating(s) or a check box with a basic reason.

2. Would a positive system that only allowed users to flag a mod upwards be beneficial, or would it require too much administration/policing to prevent someone from launching a "ZOMG PROMOTE MY MOD111!1" campaign?

Does that mean a user could only rate a mod as positive or just not at all? That would probably confuse me as a lot of users may just choose to not rate their mods while authors would be wondering what's wrong with their respective mods.

3. How else can we work to spread the news about the exciting user interfaces that are available on WoWInterface other than the current methods which show the top favorites, new addons, top downloads, our addon spotlight on the addons tab and our weekly picks, etc.?

The rating system can be a good thing but it would actually require some honest and constructive criticism from users. I would hope that all users would rate mods to help authors understand various perspectives and to be able to learn what people want... to hopefully be able to make their mods even better. Maybe we could setup a poll to ask users if they would honestly rate mods if we had a scale of like 1-5 (1 being the highest rating for this example) if they had to list a reason to be able to rank a mod at a 4 or 5? What I would still fear is that users would just rank it a 3 just so they wouldn't have to give input and possibly offend someone.

[p3lim]

Having Top Favorites, Top Downloads, the spotlight and the newest updates/uploads pretty much covers it, I can not see any need for a rating system, 'cause the favorites pretty much covers that, in a cleaner way (you dont have 'vote my addon!' crap with favorites)

my 2 cent

[Mikord]

Honestly, if you really want to examine it, all of the current systems have their issues (downloads, ratings, favorites, addon spotlight).

I'll go over some below, but the main point is that none of them are perfect, and in the end there is not a compelling reason to get rid of any of them (except maybe downloads). The categories cater to different aspects and when all taken together help provide a fairly decent picture.

Downloads

Of all categories mentioned, this one is the worst. They are really quite pointless because a mod that is frequently updated due having dynamic data will naturally get more downloads as the result of large amount of updates versus another mod that is rarely updated because it is either customizable enough that changes are rarely necessary or the function it provides simply doesn't need to change much.


Ratings

You've already touched on some issues with ratings, and there is really no need to rehash them here. While they certainly have their share of issues, I find them to be better than the current alternatives. Users should have a method to show their appreciation for the quality and usefulness of a given mod.


Favorites

This is more of use to a user who wants to get notified of updates (or have quick access add the mod to the UI updater) as opposed to any kind of rating system. For those users who don't really care to stay on the bleeding edge, there is little incentive for them to make a mod a favorite.

This also indirectly suffers from the same issue as downloads in that mods that are more frequently updated due to dynamic data are far more likely to be marked as a favorite for the notifications of dynamic data updates versus one that doesn't need to be updated often.


Addon Spotlight

While this is a good feature, it is either random or subjective to the relatively minor number of people deciding on which mods should be in the spotlight. It has little, if any, bearing on what the user community as a whole thinks of a mod.

[Dreadlorde]

I personally never look at the rating of addons, mostly because of what Cairenn described. I also think top favorites, new addons, top downloads, addon spotlight and weekly picks do everything authors need to get word out about their addon(s).

[SeiferTim]

I think that an actual rating system is the way to go - there are just too many addons out there, and a lot that seem to just be poorly made/rarely updated, etc.

What about a system where you can either give an addon a + or a - rating.
The effect on the overall rating of that addon would be based on your own 'rank', which is determined by the number of posts in the forum you have and/or the average ratings of your own addons, number of different addon downloads, etc. Unregistered users would not be able to rank any addons.

My thinking is that a user like Tekkub, who would probably have a very, very high rating, if they rate an addon as good, it should mean more than some random user who's only ever downloaded one other addon, and has never posted on the forums.

[Petrah]

Originally Posted by p3lim Having Top Favorites, Top Downloads, the spotlight and the newest updates/uploads pretty much covers it, I can not see any need for a rating system, 'cause the favorites pretty much covers that, in a cleaner way (you dont have 'vote my addon!' crap with favorites) my 2 cent

What he said.

[littlebuddha79]

The way I see it is there is one MAJOR problem being dealt with here: People are viewing the rating system as a popularity contest instead of an actual critique or view on what people think of their addon. Since there is no real way to tell who is voting just to put numbers on a particular addon (what with the anonymity of the internet and all) you can't really stop this behavior short of removing the rating system all together.

The only thing that will be accomplished by making the rating system more complicated or more in depth is to ensure that only 2 groups of people will vote: the very few that actually care and take time to rate things fairly, and the many that the authors get to vote for them to improve their rating.

As long as people on the site continue to act this way...there is no solution.

There are alternatives, but no real solution to the problem.

You could make it a requirement to be a member of the site before users are allowed to rate addons...this way you can keep track of what addons each user downloads....this list could be reset or each addon on the list could be reset every time that addon is updated
example: I download Trinity2.0. Trinity2.0 is then added to my list of addons I have downloaded. When and update for Trinity2.0 comes out, it is moved from my downloaded list to a OutOfDateAddon list. This lets you know that I downloaded it once, but have not gotten the update. If I then return to download the update, it will be required that I provide a rating of the addon. If I do not return for the update, then no rating is provided/required/allowed.
This way only return users are allowed to vote on addons, making sure that the rating is given by a user that has used the addon and has come back and clearly likes it. Also, if the user does not return for the update, then it can be determined that the user didn't like the addon enough to keep using it.

Also, if it is a requirement to rate the addon in order to download the update (for registered users, since it wouldn't be fare or practical to think that everyone that downloads from WOWI is registered) we could make the ratings A LITTLE BIT more in depth. Possibly giving a 1-5 or 1-10 rating in several (but not too many) categories like: Author feedback, Timeliness of Updates, Ease of Use, Computer CPU/Memory Hogging, etc...

This would require a revamp of the rating and the whole downloading system and would require a cache of a list of addons that every registered user has downloaded (possibly being wiped or reset, or removed from the list after a period of time), but is the best solution/alternative to the current rating system that i can think of.

And now MidgetMage55 is not the only non-author to take an interest in this

I look at ratings, but it's not the only think I look at when deciding what addon to try out.

[Tekkub]

Originally Posted by littlebuddha79 People are viewing the rating system as a popularity contest instead of an actual critique or view on what people think of their addon.

Aye, and a decent critic/recommendation system would solve this. Currently the "weekly pick" system just says to users "these are recently created/updtaed addons we like". But if we have multiple critics with short profiles describing what they look for in an addon then all bases could be covered. One critic may look for low resource use and clean code (like me), another may recommend "pretty" addons, and another might look at how intuitive the addon is to use and configure. If an addon gets a badge from multiple people, that just shows that it's really good.

[Thaoky]

Originally Posted by littlebuddha79 Possibly giving a 1-5 or 1-10 rating in several (but not too many) categories like: Author feedback, Timeliness of Updates, Ease of Use, Computer CPU/Memory Hogging, etc...

I agree with the concept, but only if it avoids some obvious pitfalls:

Author feedback: Sounds good, judging how well the author can provide feedback is interesting.
Timeliness of Updates: This one is subjective, during some periods, an author may be able to update his addons with cool new features twice a week.. then take a month or two before releasing somehing new, the addon isn't dead, but life has its load on all of us..

Ease of Use: Sounds good too.
Computer CPU/Memory Hogging: Too subjective too.. I don't want to start a debate about this, but everybody has his opinion on what a "bloated" addon is. Some people think that and addon that eats 5mb of memory is fine, some other find it bloated.. but if the addon manages a lot of data, then it's simply normal, and it's not necessarily poorly coded.

Each of these categories is just a prone to being manipulated as the current rating system.

All in all, I like the current system, and I find it nice to know how many times my addon has been downloaded. It's not a popularity contest, but it's cool to see why we're doing it. For instance, if I were to start a new project, and if only 20 people downloaded it.. then I'd probably stop it quickly. I think that at least authors should be able to have access to this info about their own addons. (same goes for favorites)

As for the best rating system, I don't know, really, maybe the best thing is simply to drop it completely, and to turn the site into more than a repository.

Promoting addons differently, and maybe more often could be nice. For instance, I could imagine that every week, one "big" addon could be introduced (like omen/x-perl/etc..) and a few smaller ones that answer a specific user need could be put in the spotlight.

Thanks for your efforts, no matter what you decide to go with

[ezarra]

The trouble with ratings is a lot of mods can't realy be rated. Eitehr they work, or they don't. I have on mod that all it does is hide the duel frame. There is no ui, no options, no nothing. It's a 1 line LUA. It's one of my favorite mods because I think duels are just about the stupidest thing in WoW (and no, they are not good practice for PVP or Arenas). Never done them, never will. How do I rate that?

I tend to ignore ratings when I look for mods because some of the worst mods I've ever run have had very high ratings, and some of my favorite mods have had very low ratings.

My current addon (I wrote a few back before all the LUA changes but those are long gone) is very simple and really doesn't do much, but it has been very useful to me on several occasions, including today when I went to get the chicken egg in Westfall. binding the chicken emote to a key allowed me to spam that emote the 100 or so times it took to get the target chicken to like me.

But on the other hand, my addon is really just a slightly more convenient way of spamming emotes than making a macro. Would I rate it 5 stars? Well, for me it's that useful. Do I think it's worth the same rating as Cartographer? Well, no, not even close. Autobar? FuBar? No, those all make fundamental and drastic changes to the way I am able to play the game. I don't even think of them as the same class as little addon hacks like IgnoreDuel or EmoKey or AutoRez.

It's almost like there should be different classifications of addons. Hacks, addons, and Meta-addons. (Cartographer, Auctioneer, Bigwigs, etc would be Meta-Addons).

[Zyonin]

I usually don't look at ratings when I grab mods, I tend to look at descriptions, comments, favorites. If you really want a rating system, then take a look at Slashdots comment rating system.

A group of reviewers might be an idea. Reviewers would be a groups of users (likely users who have been here a while) who review mods then write up reviewed based on if the mod acutally functions, does it do what it says it does, look and feel of the mod, etc. Just my two cent's worth.

[Voorije]

Im sorry if this has been mentioned before, I didn't have time to read all the posts (lots of walls of text) but in regards to ratings, if you do implement them there should be some kind of timeout, or individual version ratings.

Addons improve over time, a new fledgeling addon may be poorly written, buggy, or just not as usefull as people would like, this may cause a number of low ratings.

As time goes on authors improve thier addons and shape them into something more honed to a specific purpose, fufilling the niche that everyone wants. Old poor ratings can effect an overall score and i don't think it's representative to cound old ratings from long ago.

If you do implement something, I'd like to see a solution to combat this problem.

Perhaps adopt an idea like comments on wowhead.com where each comment is tagged with the version of wow at the time of posting. Perhaps a similar idea, like the version of the mod can be tagged to each review/rating and publically viewable?

[Recluse]

While the rating system which was in place could certainly be abused - ANY system has the same potential.

Someone had mentioned that perhaps a commentary of some sort should be required to submit a rating, and while I agree that this should take place, who would be there to police what was being said, if anything? "Oh, 100 words minimum required." *Edit>Select All>Copy>Paste*. Not to mention it being one other place Authors would have to respond at. There's already a comment section for the addon in question, no need to double up.

I also saw mention of quantity of downloads / number of times favorited, but this is also abuse-prone. If I wanted to buff my addon's counts, it would be quite simple to do so, and I'm sure it would be the same for others as well.

The only thought I had was something more along the lines of detecting outliers and such, but over time, the abusers would outnumber the actual ratings with their outliers, resulting in a sway of ratings as well.

In the end, I'm left empty of ideas for a rating system which would be foolproof, but I didn't stay up late brainstorming.

Myself, I did keep an eye on my ratings, but it wasn't because I wanted to compete with other addons, but it was a judge of audience satisfaction is all. When I go looking at other addons for something I might like to use, I don't look at others' ratings, as the rating isn't important IMO. The addon's features and so on are the important bits. Just because an addon is popular, it doesn't mean I would, want to, or be influenced to try it out.

Perhaps if there were a more private user-to-author rating system, which meant nothing to popularity listings, it would be abused less, but still provide the feedback to the author?

Originally Posted by kerrang I know that many addon authors are a bit sick of the way users will talk about 'light' addons - "memory efficient" addons etc. - when in truth they have NO idea what they're talking about because they simply don't understand what does and does not make a well written/efficient addon. The idea that you would publically 'benchmark' an addon is just going to encourage this mentality IMO - and I'm not sure who would benefit other than the sort of people who think 'their' way is the only way (e.g. code nerds!) I really don't think we need to encourage people caring about the technical side of it...

I don't see it as possible that these technical reviews would be unbiased. They would have to be extremely limited and precise in what they were testing, and even then, I don't see how the environment would be controlled enough to do so accurately. Remember, these benchmarks would be done by people who are probably addon authors themselves, and likely running their tests on some spare computer at home.

Second, I don't see how much of the information gathered would be of use to anyone when deciding which addons to download. Simply because one addon does similar things to another does not mean that they should have the same benchmarks. Perhaps one of them has a few small features which makes its benchmarks slightly higher, but acceptably so. Because most users WILL look and judge an addon by these numbers, even if they have no idea what they really mean, they'd pick up the one with better numbers, regardless of the features. This could be a good thing, but not consistently so.

As Tekkub mentioned, I agree that it would help to "encourage good coding practices", but some addons, even with perfect code, could be highly feature-bloated and appear to have bad tech ratings because of this.

I just don't think that such a feature would be subjective enough, and definitely misinterpreted by the majority of non-author users. If the users "don't understand what's under the hood", then why bother showing them the engine's specifications, as if it would help them pick their addon?

I don't mind such a project being an external thing, but integrating it in places such as this website would just result in confused average-joes and an addon author e-peen competition.

[eiszeit]

Originally Posted by p3lim Having Top Favorites, Top Downloads, the spotlight and the newest updates/uploads pretty much covers it, I can not see any need for a rating system, 'cause the favorites pretty much covers that, in a cleaner way (you dont have 'vote my addon!' crap with favorites) my 2 cent

What he said.

I also like what Taffu said. Something like a "Badge" system where the author can mark some points like "easy to configure" etc..

[Tekkub]

I always rated my addons as a 5 when I submitted them. Not really to "game" the system, but simply because people so rarely rated addons that when someone did, it was often a bad rating because they didn't like/understand the addon. All in all, community based ratings don't work well and encourage the exact things you stated.

But at the same time, some means of gauging "popular" addons besides the basic download/favs stats would be good. Downloads/favs only favor the biggest addons, or the ones that get a userbase that update and then pushes out super frequent updates.

Maybe something along the lines of a "social network" where you can watch the authors you like for updates, watch friends for changes to their favs and addons they download the first time.

Heck, maybe even come up with a subset of the "pick of the week" where certain users can give their "badge of approval" to an addon on some interval, like say one per week. The addon's pages could also have "Pick of the week <date>" and "Recommended by <user>" badges added to them as well. Naturally these "critic" users wouldn't be able to rate their own works, and should be discouraged from rating projects they've been involved in. Something similar to the featured authors... featured reviewers. Heck, you could even require a short writeup to be required when recommending an addon.

And, above all, don't allow for negative ratings of any sort. But don't discourage CONSTRUCTIVE feedback as well.

[Mike-N-Go]

Here are some ideas to revamp the rating system, basically, I did a /braindump, but I hope you can understand me.

First off, a 1-5 rating, as it was in the past, did not really help me, as one seeking addons, know much about it on 1-5 scale? Of awesomeness? Awesomeness is relative. One alternative would be a "Would you recommend this addon to others?" and a "yes" and "X-Mod instead", etc. with optional or required comments. The site could even have respective tabs for this on addon pages, with the recommendations helping people understand their mods. Also, a running ticker that has the category of "Recommendations" could be placed when viewing the WoWInterface Downloads. The problem with the old system was it did not allow one to see how many votes placed at what amount, that averaged all the votes; my suggestion allows everyone to voice their opinion either way, if someone votes it down, and does not give a meaningful comment, then I could see they made it purely for stupid reasons. If someone leaves constructive criticism or comments, then I could better understand their views of a specific addon. Additionally, this could be extended to show the recommendations under the author's profile, so one could view feedback to said author. Also, rating feedback by "Was this comment helpful to you?" yes/no per a user's feedback could order the list of feedback. "Report this feedback as unconstructive?" yes/no could be added as well, allowing other users to help tag feedback that is damaging or degrading, if some number is reached, it could hide the comment from view unless someone wants to click on it to view it, and a further number, an admin could be alerted of it and then remove it if he sees fit. This feedback could also be considered as 'reviews' for the really thick, multi-parigraph comments. Highly ranked reviews could even be feautured. This system is, more or less, a sub-category of the downloads, allowing those that download, to make comments. My vision should require commenters to have a WoWInterface account, and should alert the user if they have not commented on whatever addon. Additionally, each feedbacker could have shown publicly their history how they rated mods, so those who want to understand their comments better. Also, users should be able to update their comments, though this might undermine the feedback on feedback feature; this could be rest whenever someone updates their feedback. A category "User Recommend" could be added for the top recommended addons on the main page, also, this would be yet another way to sort addons, though, should account for total recommendations to time available, with the default available counted as a month, to better prevent new addons from shooting to the top of the results. My idea would be yet another way for the community to get involved. I believe I have thought of a system to prevent fake or unhelpful feedback, though, the problem remains of getting most beyond downloading a file, and to give constructive, helpful criticism of a mod.

[Thaoky]

@Tekkub & littlebuddha79: I agree with your answers to my post, I didn't elaborate because as I said, I didn't want to go off-topic as it belongs to another debate.

Come to think about it, maybe such criterias would be worth being displayed if the rating is set by a "committee" of guys who can make an educated judgment on the quality of addon. I'm ok to hear that the quality of my code or of the structure of my db is only worth 3/5, but only if 1) I know why, and 2) that rating is given by someone who really knows how to judge based on measurable criterias.

I would hate to be judged on these points by non IT-literate users. I would, however, appreciate the feedback (and possibly rating) of non IT-literate users on things like user friendliness, ease of use, etc... But this feature and the previous one are actually roles that are partially filed by a forum, as these issues are more likely to be addressed by a constructive debate rather than by a simple rating.

It's a tough choice to decide how to effectively address this problem if you ask me

[littlebuddha79]

Originally Posted by Thaoky @Tekkub & littlebuddha79: I agree with your answers to my post, I didn't elaborate because as I said, I didn't want to go off-topic as it belongs to another debate. Come to think about it, maybe such criterias would be worth being displayed if the rating is set by a "committee" of guys who can make an educated judgment on the quality of addon. I'm ok to hear that the quality of my code or of the structure of my db is only worth 3/5, but only if 1) I know why, and 2) that rating is given by someone who really knows how to judge based on measurable criterias. I would hate to be judged on these points by non IT-literate users. I would, however, appreciate the feedback (and possibly rating) of non IT-literate users on things like user friendliness, ease of use, etc... But this feature and the previous one are actually roles that are partially filed by a forum, as these issues are more likely to be addressed by a constructive debate rather than by a simple rating. It's a tough choice to decide how to effectively address this problem if you ask me

This would acually work fairly well, and could be easy to impliment. The rating for the resource use/make up of the addon could be limited to registered Authors. General users would not have access to that particular rating and only other authors could vote on it.