After my last two posts, I was talking to my roommate. I think this was his suggestion, if not a direct descendent of his suggestion, but he said to just try playtesting it myself. For some reason this hadn't occurred to me, but it made perfect sense. What I am doing now is just that: I'm making groups of characters with a single ability battle each other.
I decided that, because the game is going to so rarely be about one-on-one, to test a 3-on-3 game. For all of my examples, I have had three characters with bows ('archers') fighting three characters with some kind of stat increase ('soldiers'). Basically, the characters would march forward as the archers were shooting them. I recorded how much damage they took before getting into melee, and then the end result of combat.
So far, my predictions have been mostly true. If a party has a high dodge, they dominate. If the have a high armor, they still win with some casualties. If they have an extra attack, it ends in a draw.
I wish I had thought of this before (and can't really believe I didn't). I see this as kind of the gaming equivalent of a lab experiment: everything is done under perfectly controlled (and closed) settings, and you can test one thing really well. What I'm going to start doing is testing how well different attacks fare against each other. For example, next I'll have a crossbow and see what happens, then I'll start using special attacks like "Life Drain," or even summon powers, and seeing who wins.
This has left me with two frustrations: First, this is something that I could do much faster if my computer was working better. Really I could just put all of the parameters in, then run it 1000 times, and see how often the soldiers beat the archers (instead of, "Yeah, I played 2 or 3 games, they seemed to go like this"). Maybe it's worth seeing if I could get Python working or something. Second, I discovered this magical development tactic two days before I move out of my current house and three days before I take a trip across the country. :( Oh well...