Wednesday, July 6, 2011
By: Matthew Doucette
Decimation X3's results of two rating manipulation attacks.
The first attack starts off-screen, as data was collected only after the first attack started.
I hate writing articles like this. :(
But, I have to document it for reference purposes.
It is now twice that our games have had their ratings abused on Microsoft's Xbox LIVE Indie Games platform. The first occurred in March 2011, detailed in Michael Neel's "The Real March Madness - XBLIG Rating Manipulation Continues" article. The second occurred in June 2011, as brought to light by my APP HUB posts (1,2,3), all three quoted here:
APP HUB Posts:
Charles,
There is now more rating abuse taking place.
Our ratings are dropping in Japan as well as beginning to drop in the U.S. again (and presumably in other countries). Only for Decimation X3 and Score Rush, but not Decimation X (our lowest rated game before the abuse):
So now our worse game is #3 in Japan and the others are #21 and #35... they used to be #3, #4, and #5.
http://marketplace.xbox.com/ja-JP/Games/XboxIndieGames?SortBy=TopRated
And now our worse game is also the highest in US, where it was left after the first rating abuse, and the others are #220 and #221... they used to be in the top 30. They were approximately #170..#180 after the first rating abuse.
http://marketplace.xbox.com/en-US/Games/XboxIndieGames?page=8&sortby=TopRated
It's hard not to view this as a lack of respect for your developers. Can you please fix this, for the sake of doing what is right for your developers? What do we have to do to inspire you to fix what rightfully should be fixed without needlessly challenging our relationship? I have not gone public with this which has been my overwhelming feedback on what to do, to essentially force Microsoft in a position to do the right thing. I don't even want to do such a thing. I support Microsoft, this is clear. Please support me back.
Thank you.
Matthew Doucette, Xona Games
- http://forums.create.msdn.com/forums/p/80881/516323.aspx#516323
I'll show the example of just Score Rush in Japan:
Score Rush 28/06/2011 4 194 24 (rating, number of votes, rating rank)
Score Rush 27/06/2011 4 194 25
Score Rush 26/06/2011 4 194 28
Score Rush 24/06/2011 4 194 28
Score Rush 23/06/2011 4 194 29
Score Rush 22/06/2011 4 194 30
Score Rush 21/06/2011 4 194 30
Score Rush 20/06/2011 4 194 30
Score Rush 19/06/2011 4 194 31
Score Rush 18/06/2011 4 193 29
Score Rush 17/06/2011 4 193 28
Score Rush 16/06/2011 4 193 28
Score Rush 15/06/2011 4 193 28
Score Rush 14/06/2011 4 193 30
Score Rush 13/06/2011 4 193 30
Score Rush 12/06/2011 4 193 30
Score Rush 11/06/2011 4 185 7
Score Rush 10/06/2011 4 185 7
Score Rush 09/06/2011 4.25 177 6
Score Rush 08/06/2011 4.25 177 6
Score Rush 07/06/2011 4.25 176 6
Score Rush 06/06/2011 4.25 176 6
Score Rush 05/06/2011 4.5 158 3
Score Rush 03/06/2011 4.5 158 3
Score Rush 02/06/2011 4.5 158 3
Score Rush 01/06/2011 4.5 158 3
Score Rush 31/05/2011 4.5 158 3
Score Rush 30/05/2011 4.5 154 3
Score Rush 29/05/2011 4.5 154 3
Score Rush 28/05/2011 4.5 154 3
Score Rush 27/05/2011 4.5 154 3
Score Rush 26/05/2011 4.5 154 3
Score Rush 25/05/2011 4.5 154 2
Score Rush 24/05/2011 4.5 154 3
Score Rush 10/05/2011 4.5 150 3
Score Rush 09/05/2011 4.5 150 3
Score Rush 08/05/2011 4.5 150 3
Score Rush 07/05/2011 4.5 150 3
Score Rush 06/05/2011 4.5 150 3
Score Rush 05/05/2011 4.5 150 3
Score Rush 04/05/2011 4.5 150 3
Score Rush 03/05/2011 4.5 150 3
Score Rush 02/05/2011 4.5 150 3
Score Rush 30/04/2011 4.5 148 3
Score Rush 29/04/2011 4.5 148 3
Score Rush 28/04/2011 4.5 148 3
Score Rush 26/04/2011 4.5 148 3
Score Rush 25/04/2011 4.5 148 3
Score Rush 24/04/2011 4.5 148 3
Score Rush 23/04/2011 4.5 148 3
Score Rush 22/04/2011 4.5 148 3
Score Rush 21/04/2011 4.5 148 3
Score Rush 19/04/2011 4.5 148 3
Score Rush 18/04/2011 4.5 148 3
Score Rush 17/04/2011 4.5 147 3
Score Rush 15/04/2011 4.5 146 3
Score Rush 14/04/2011 4.5 146 3
Score Rush 13/04/2011 4.5 146 3
Score Rush 12/04/2011 4.5 146 3
Score Rush 11/04/2011 4.5 146 3
The math fits that every vote after 158 (after June 6th, 2011 and the end of our #3 spot in Japan) could have been a 1 star vote. It only takes 36 such votes to do this damage. 1 star votes cause incredible damage to top rated lists.
Matthew Doucette, Xona Games
-http://forums.create.msdn.com/forums/p/80881/516374.aspx#516374
"Murudai" posted:
...the position on the Top Rated list does actually make a big difference.
This is information Robert Boyd has already clearly demonstrated to us, and I appreciate you coming forward, Murudai, to silence any doubt.
The most scarring issue is the indirect value of discrediting the developer. In our talks with publishers and media we mention our #3/4/5 ranks of 1,800 games in Japan all the time. It doesn't help that http://marketplace.xbox.com/ja-JP/Games/XboxIndieGames?SortBy=TopRated shows Decimation X3 and Score Rush at #20 and #39 (at the time of this writing). Our ratings is one of our standout accomplishments, beyond our $100,000 contest win, as it's a measure of our success on the field. Damaging to Microsoft is where the conversation leads to next, which is talks of rating manipulations and the lack of support. I wish I could avoid this all.
What about the value of simply doing what is right, respecting (top) developers, and showcasing truthful lists to the Xbox LIVE gaming community?
Matthew Doucette, Xona Games
- http://forums.create.msdn.com/forums/p/80881/520197.aspx#520197
Stat Screenshots:
Michael Neel of GameMarx and Xbox Indies, both great sites, helped shed light on the both issues with stat tracking. Here you can see the stats of all our games and how they were targeted and abused via rating manipulation:
- xboxindies.com/game/score-rush/stats (targeted)
- xboxindies.com/game/decimation-x3/stats (targeted)
- xboxindies.com/game/decimation-x/stats (potentially targeted)
Here are screenshots of each, as of today (July 6th, 2011):
Score Rush stats, showing rating manipulation.
Look at the "Top Rated Position" graph for United States and Japan.
Source: xboxindies.com/game/score-rush/stats.
Decimation X3 stats, showing rating manipulation.
Look at the "Top Rated Position" graph for United States and Japan.
Source: xboxindies.com/game/decimation-x3/stats.
Decimation X stats, showing rating manipulation.
Look at the "Top Rated Position" graph for United States (potentially targeted) and Japan (not targeted).
Source: xboxindies.com/game/decimation-x/stats.
.
Please note, these stats do not cover the beginning of the first rating abuse. They were collected only after it started.
Ranks Before and After:
After both rating abuse manipulations, as of today (July 6, 2011), we had two top 30 games in the U.S. kicked down to 200, off the charts essentially. And in Japan, we had the #3, #4, and #5 games, two of them kicked down to #20 (Decimation X3) and #39 (Score Rush). The untargeted and worse of our games jumped from #5 to #3 with the absence of our other two games (Decimation X). This is out of 1,800 games These changes are signifcant.
Conclusion:
And Microsoft's official response to all of this: "We have also investigated rolling back suspect votes, however, we determined it will not be possible to do this." - Charles Cox, Microsoft. Of course it is possible to do this, and I wish they would.
There are so many issues. One is the lost ability to rate games in the Games Library on the Xbox 360 Dashboard, which has lowered the amount of XBLIG voting that takes place which makes rating manipulations easier to perform. I created a connect issue on it which currently has an incredible 105 up votes and 0 down votes, but no response from Microsoft. Please vote and comment on it, it requires a free signup to Microsoft Connect:
It's not like Microsoft never moves. So don't read this the wrong way. They reacted to the most up voted connect issue on XBLIG ever, when XBLIG devs fought to move Indie Games back under "Games and Demos:
There's a lot of dreams I have for XBLIG, including the "Xona Proposal", that may never see the light of day. It's a truly amazing platform with even more amazing potential. It's sad to see it only get half way there, because the second half could truly be magnificent for indie game developers and gamers alike.
About the Author: I am Matthew Doucette of Xona Games, an award-winning indie game studio that I founded with my twin brother. We make intensified arcade-style retro games. Our business, our games, our technology, and we as competitive gamers have won prestigious awards and received worldwide press. Our business has won $190,000 in contests. Our games have ranked from #1 in Canada to #1 in Japan, have become #1 best sellers in multiple countries, have won game contests, and have held 3 of the top 5 rated spots in Japan of all Xbox LIVE indie games. Our game engines have been awarded for technical excellence. And we, the developers, have placed #1 in competitive gaming competitions -- relating to the games we make. Read about our story, our awards, our games, and view our blog.