Wow, people really like Badges! My mahjong game (3D Mahjong Solitaire) has been out there for a month or so with just a “free play” mode, and had received a bunch of very positive reviews. Last week I added “Champion Mahjong” mode, in which you complete tables to acquire Badges (similar to Xbox LIVE achievements). Unfortunately, the tables as they were defined were way too difficult – e.g. requiring 100 tries to pass in some instances. The net result is that everyone got stuck on the few tables, and the negative reviews began pouring in (each one = pain).
I put a bunch of effort into generating “guaranteed winnable” layouts – and pretty much failed the first time. The problem was that I wasn’t testing the results, and doing to much assuming. So, I got creamed in the reviews again last night/today (ouch again 
). The other challenge is that building guaranteed winnable mahjong tables is actually a pretty tricky problem; everything I could find on the internet talked about “just play the game backwards” – which it turns out is a load of tripe. More on that in a moment…
So, I took a step back and created a “headless” version of the Mahjong code which could automatically play all of the tables a bunch of times, and allow me to do some statistical analysis on the win/loss ratios of them all. Here’s what I saw with the currently posted “guaranteed winnable” logic after playing through every table 100 times each:
Putting that into English: many of the tables were completely unwinnable! OhmigodDOH moment for sure. And this is after I posted a comment and an update saying that the problem had been fixed. No wonder people were flaming the game in the comments; I was half-tempted to do so myself! ><…
So, back to the “guaranteed winnable” algorithm: after playing and playing and playing around with the “play it backwards” approach, I increased the overall win/lose ratio by a whopping 3%. Yeah, that would stop peoples’ frustration . The problem is that the “guarantee winnability by playing the game backgrounds” approach doesn’t guarantee winnability. Simplest example is if you had a layout with 4 horizontally arranged tiles, you could start by placing a pair of matching tiles in the 1st and 3rd positions; and you’d be left with the 2nd and 4th positions for other pair of tiles – resulting in an unwinnable board.
I fought with this left and right before taking a step back and realizing that there was a much better approach: play the game “forward” (like a human would), picking any two “free” tiles and ignoring their tiletype; then just remove those tiles from the board, set those tiles to matching types, and continue on your merry way until the table is empty. At that point, you just put all of the tiles back on the table with their new matching styles.
When trying the “play it backwards” approach, the “GetAvailableTile” function grew to 100 lines of gobbledegook. But with a “play it forward” model, it’s dead-simple:
static Tile GetAvailableTile(Tile dontmatch)
{
int offset = RandomMgr.Next(RemainingTiles.Count);
for (int i = 0; i < RemainingTiles.Count; i++)
{
Tile tileToRemove = RemainingTiles[(offset + i) % RemainingTiles.Count];
Debug.Assert(!tileToRemove.Removed);
if (dontmatch == tileToRemove)
continue;
if (tileToRemove.IsFree)
return tileToRemove;
}
return null;
}
So now when I run the analysis tool, I get an average win/loss ratio of 45% which is MUCH better:
Wonderful! Now a “dumb” computer can solve almost all of the puzzles in just a few tries…
Next up, intuitively you’d want the tables to get progressively harder. Now that I’ve got the analysis tool, I can just sort by the Win/Loss ratio in Excel:
And now I know what order to place the tables in so that it gets harder as you go. Furthermore; when I add a new table, I can simply run the tool against it and tweak the table so that its difficulty is at an appropriate level.
Finally, you’ll notice that the last few levels are still too difficult; indeed, Table 21 was impossible for the “dumb” computer to solve! So now I have the knowledge and tools with which to go back into the layout editor, tweak those layouts until they’re closer to a ~.15 win/loss ratio (what I’d consider appropriate for the hardest levels), and then the whole string of tables should be doable, the earlier badges will be easier and the later badges should be harder to get (as they should be), and everyone goes back to spending time enjoying the game instead of posting frustrated comments 
.
So: morale of the story: Don’t assume; statistical analysis rocks. When 7YRL gets to the “level balancing” stage, I’ll be doing a heck of a lot more of this at that point!
(Oh, and finally: you might ask why I’m spending all this time writing this post when I haven’t actually submitted the update yet. It’s because MS’s submission process – which in almost all ways rocks – sucks when you’ve got an update in the queue that you want to override. I need to wait for that update to go through, and then can submit this fix. So, more waiting – and more frustrated players in the meantime… ><…)
UPDATE (4/17 @ 12:49PM PST)
I’ve just completed the work described above. This required reordering the tables and making some tweaks to the tables with too-low win/loss ratios. After all of that was done, here’s what the new stats look like:
Things to note:
- All Win/loss ratios are at .15 or above. My assumption is that a good human player can probably double win the rate of a “dumb” computer, so the harder levels will take them about 3 tries to pass. That’s about right, and leaves me some leeway to add harder levels without getting too hard.
- The table difficulties smoothly decrease, meaning that the earlier levels are easier and the harder levels are harder (as you’d expect).
- There’s three exceptions to the smoothly-decreasing: Tables 8-10 get really easy. Although I could have left them up at the front, I opted to move them in a little ways because the table layouts are pretty sparse and not a good introduction to the gameplay. Over time I can look at increasing the difficulty of those levels so that they fit in better difficulty-wise. But the user won’t really be able to tell either way…
So, now the waiting begins for MS to publish the update that they’re currently testing, and then I can submit …
To those of you coming here from my comment in the Wp7 Marketplace review for my game – thanks for your patience, and please up-rate your rating once the new checkin goes in! 
Cheers,
Jeff
.
