Skill Challenge Analysis
Dungeons & Dragons 4th edition includes the concept of "skill challenges," a multi-skill check test. It might abstractly represent a chase through a busy market, negotiations with a lord, or an interrogation. I love the idea. Properly done, it could give non-combat aspects of characters a place to shine. Unfortunately it wasn't properly done.
I've run the numbers for the system, both as published, and applying the "Update version 2" dated "7/2/08" from Wizards. Official link, archived link.
Reading the tables
"Skill Mod-DC" is the total skill modifier brought against the check minus the DC of the check. This is completely independent of how that modifier is achieved.
"Per check chance" is, given the previous column, the chance of succeeding at a single check.
The "Challenge Difficulty" columns are for the 5 levels of challenge difficulty. The coding "4S vs 2F" means "4 successes before 2 failures." The result is the percentage chance of success for the challenge overall.
"Expected levels" is at which levels you can expect this "Skill Mod-DC". This assumes the character has 18 in the relevant stat at first level, that they always increase the stat whenever possible, that they are trained, but that they have no further improvements. The DCs are the "medium" difficulty for a given level, including the +5 modifier to skill checks. These assumptions match, and provide the same answers, as Stalker0's excellent analysis.
These numbers are all exact (to the precision listed, so plus or minus one tenth of a percent). The percentages were taken from the absolutely exact probabilities tracked as fractions. These are not estimates, nor the results of a Monte Carlo simulation. They were calculated with custom software I wrote.
The only assumptions are the last column, "Expected levels." If you don't like Stalker0's and my assumptions, feel free to scoot the levels up or down until you're happy. If you think we should assume that 4 other party members are using "Aid Another" to add +2 each for a total of +8, move the levels up 8 rows.
As Published
Note that this includes a +5 to difficulty classes for all checks. This matches the rules as written and to the best of my understanding. If you disagree with that interpretation, you'll want to move the entries in "Expected levels" five rows up.
| Challenge Difficulty | |||||||
|---|---|---|---|---|---|---|---|
| Skill Mod-DC |
Per check chance |
1 (4S vs 2F) |
2 (6S vs 3F) |
3 (8S vs 4F) |
4 (10 vs 5F) |
5 (12 vs 6F) |
Expected levels |
| -20 | 5% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | |
| -19 | 10% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | |
| -18 | 15% | 0.2% | 0.0% | 0.0% | 0.0% | 0.0% | |
| -17 | 20% | 0.7% | 0.1% | 0.0% | 0.0% | 0.0% | |
| -16 | 25% | 1.6% | 0.4% | 0.1% | 0.0% | 0.0% | |
| -15 | 30% | 3.1% | 1.1% | 0.4% | 0.2% | 0.1% | |
| -14 | 35% | 5.4% | 2.5% | 1.2% | 0.6% | 0.3% | |
| -13 | 40% | 8.7% | 5.0% | 2.9% | 1.8% | 1.1% | |
| -12 | 45% | 13.1% | 8.8% | 6.1% | 4.3% | 3.0% | 7 |
| -11 | 50% | 18.8% | 14.5% | 11.3% | 9.0% | 7.2% | 1, 4, 5, 10, 11, 13, 19 |
| -10 | 55% | 25.6% | 22.0% | 19.1% | 16.7% | 14.7% | 2, 3, 6, 8, 9, 12, 16, 17, 20 |
| -9 | 60% | 33.7% | 31.5% | 29.6% | 27.9% | 26.4% | 14, 15, 18, 21, 22, 23, 25 |
| -8 | 65% | 42.8% | 42.8% | 42.6% | 42.3% | 42.0% | 24, 26, 27 |
| -7 | 70% | 52.8% | 55.2% | 57.0% | 58.4% | 59.7% | 28, 29 |
| -6 | 75% | 63.3% | 67.9% | 71.3% | 74.2% | 76.5% | 30 |
| -5 | 80% | 73.7% | 79.7% | 83.9% | 87.0% | 89.4% | |
| -4 | 85% | 83.5% | 89.5% | 93.1% | 95.3% | 96.8% | |
| -3 | 90% | 91.9% | 96.2% | 98.1% | 99.1% | 99.5% | |
| -2 | 95% | 97.7% | 99.4% | 99.8% | 100.0% | 100.0% | |
| -1 | 100% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | |
Conclusions (As Published)
That's the hard math. Now for my opinion.
I began this project because I wasn't convinced of Stalker0's math. So I set out to reproduce his numbers. I wrote my system from scratch directly using the reference books, not other people's summaries of the books. And in the end I got the same numbers. So now I believe that Stalker0's math is correct.
I also now agree with Stalker0's core belief: that the system as written is broken. Various different interpretations of the rules can change what is broken where, but ultimately the system is broken.
Your chance of success bounces up and down strangely as you level. In several cases going up a single level can cause the odds of succeeding on a given check to change by 10% (This would be levels 6 to 7, 7 to 8, 13 to 14, and 18 to 19.). This is a mess.
The system spans a number of pages, but it's still very unclear, as evidenced by the number of people chiming in with alternate interpretations. Do you apply the +5 to DCs for the individual checks? Can you generally use Aid Another to get +8 in a 5 person party? How often should PCs claim a +2 circumstance bonus? Should we use other skill DC tables? This is a mess. Given the math at the core of the system, the assumptions need to be spelled out so GM tweaks are intentional, not accidental.
As for tweaking, knowing the assumptions would help, but given the complexity of the system, without a table like the above you would be hard pressed to predict the impact of giving a seemingly small modifier. A modifier of 2 on every roll would be as insignificant as less than 0.1% or as huge as 34.5%.
No matter where you push the expected levels, a change of 10% is a huge deal. Even 5%, a single row, can be a big deal. A character who chose to start with an attribute at 16 instead of 18 can easily have a 10% worse chance failure.
Ultimately, what is the expected success rate? We have no idea. If the goal is to be like combat, with the PCs winning 95% or more of the time, you want the skill modifier to be 2 to 4 points less than the DC, and to never leave that range of 3 points. Given that the range is 7 points now, it's simply not possible. Some people would suggest that any number between 50 and 90 is acceptable; again you are limited to a range of 3 points, from -5 to -7, or maybe -4 to -7.
Ultimately, I have no idea what Wizards of the Coast expects, and that's the problem. There are a huge number of hidden assumptions built into the system. Depending on which assumptions I go with, I get wildly different results. That is why the system is broken.
Version 2
This incorporates the changes from "Update version 2" dated "7/2/08" from Wizards. Official link, archived link.
| Challenge Difficulty | |||||||
|---|---|---|---|---|---|---|---|
| Skill Mod-DC |
Per check chance |
1 (4S vs 3F) |
2 (6S vs 3F) |
3 (8S vs 3F) |
4 (10 vs 3F) |
5 (12 vs 3F) |
Expected levels |
| -20 | 5% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | |
| -19 | 10% | 0.1% | 0.0% | 0.0% | 0.0% | 0.0% | |
| -18 | 15% | 0.6% | 0.0% | 0.0% | 0.0% | 0.0% | |
| -17 | 20% | 1.7% | 0.1% | 0.0% | 0.0% | 0.0% | |
| -16 | 25% | 3.8% | 0.4% | 0.0% | 0.0% | 0.0% | |
| -15 | 30% | 7.0% | 1.1% | 0.2% | 0.0% | 0.0% | |
| -14 | 35% | 11.7% | 2.5% | 0.5% | 0.1% | 0.0% | |
| -13 | 40% | 17.9% | 5.0% | 1.2% | 0.3% | 0.1% | |
| -12 | 45% | 25.5% | 8.8% | 2.7% | 0.8% | 0.2% | |
| -11 | 50% | 34.4% | 14.5% | 5.5% | 1.9% | 0.6% | |
| -10 | 55% | 44.2% | 22.0% | 10.0% | 4.2% | 1.7% | |
| -9 | 60% | 54.4% | 31.5% | 16.7% | 8.3% | 4.0% | |
| -8 | 65% | 64.7% | 42.8% | 26.2% | 15.1% | 8.4% | |
| -7 | 70% | 74.4% | 55.2% | 38.3% | 25.3% | 16.1% | |
| -6 | 75% | 83.1% | 67.9% | 52.6% | 39.1% | 28.1% | |
| -5 | 80% | 90.1% | 79.7% | 67.8% | 55.8% | 44.8% | |
| -4 | 85% | 95.3% | 89.5% | 82.0% | 73.6% | 64.8% | |
| -3 | 90% | 98.4% | 96.2% | 93.0% | 88.9% | 84.2% | |
| -2 | 95% | 99.8% | 99.4% | 98.8% | 98.0% | 97.0% | 7, 13, 19, 25 |
| -1 | 100% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 1, 4, 5, 10, 11, 16, 17, 20, 22, 23, 26, 27, 28, 29 |
| 0 | 100% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 2, 3, 6, 8, 9, 12, 14, 15, 18, 21, 24, 30 |
Conclusions (Version 2)
It's an improvement. In the old system, highly skilled characters wanted longer challenges, while less skill characters wanted shorter challenges. Now a character always wants a shorter challenge. The difficulty dropoff is slightly less steep.
The most obvious problem is that the reasonably skilled character now has a 100% (or near 100%) chance of success. That's just dull. The skill target numbers have gone from very hard to very easy. Increasing difficulties a few points will help.
The next problem is that the difficulty slope for the longer challenges is extra steep. This seems bad. If your skills aren't quite optimal, your odds plummet.
Overall Conclusions
Now that I've had more time to reflect on this, I think I see a key problem. An important goal of skill challenges is to make non-combat encounters as exciting as combat. However, combat isn't an all-or-nothing situation. Careful strategy can massively swing the results. A single failure to hit is usually irrelevant. The characters can trade daily powers and action points to help. It's possible to recover from a bad situation. None of this is present in the skill challenge system. It's all or nothing.
Revision history
2008-06-28: Originally posted
2008-07-17: Updated with revised official rules, reorganized, added overall conclusions.
2009-01-05: Typo fixes.