Monday, March 8, 2021

What's Broken: Reviews

 On a regular cycle, someone complains about the death of the OSR (whether it was 2013 or 2017 or yesterday or tomorrow), and inevitably somebody chimes in talking about how things got “too commercial” and everybody in the “scene” wanted to make a buck and crank out a product.  That’s worth talking about, but there are some other issues that dovetail with it that I think I need to discuss first so we get a complete picture of where I’m headed with all this.



One core issues I see - and I’ve occasionally complained about it on social media, I suppose - is our lack of a review culture.  We don’t have meaningful reviews to consult and compare.  We have shilling and we have tear-down rants, and very little in-between.


There are two large classes of OSR “reviews”:


  • THIS IS BAD AND I HATE IT

    • “This is not written the way I would write it” (fair enough, but diminishing returns as to usefulness of this sort of review)

    • “This classic product sucks/can’t-be-run” (often with a side dish of “I’m much cleverer than these 70s-80s people”, which is distasteful even when true)


  • THIS IS AWESOME AND I LOVE IT

    • “Everything this guy/company does is equally great”  (usually not true, and how could it be?)

    • “This looks really high-quality doesn’t it”  (so?)

    • “This is super-innovative, subversive, artsy and I love it.  Also, my friend made it.  No, I haven’t run it LOL who actually plays games?”

    • “I used parts of this, twisted to meet my needs under a different ruleset, and I changed most of it” (interesting reading, and a standard byproduct of the way we hobby, but perhaps less useful to some readers)


With most of both of these written without ever having run the product.  Imagine how different our discourse might be if we had the expectation that once you ran an adventure, you actually talked about how it went.  Instead, we’re stuck in a rut where we have a “balance” between rants and stroking.  Bryce Lynch writes reviews which are sometimes amusing to read, but have become so repetitive as to be much less useful than they used to seem (whether this is Bryce’s fault or that of the endless stream of mediocre adventures he dares to read is a separate question).  Fear of a Black Dragon produces interesting reviews much of the time, although often they take a larger work and bang it into a rules-light shape.  Questing Beast usually does “reviews” that are more about the paper quality and the art than about using the material at the table.  Some old-school bloggers go in-depth into “what’s wrong” with a TSR module - and maybe even attempt to provide solutions - but never bother to run the adventure, even WITH the proposed fixes.  


Can’t we have reviews by people who ran the product?  Answer: perhaps not if we want reviews to appear during the main sales push cycle for a product.  Yet if everything’s available forever, I’d happily take a robust long-tail review culture over what we have now.


Here’s a call to action: if you haven’t actually used the product, don’t call it a review.  Have some dignity.  Call it a “first look”, a “preview”, an “impressions”, whatever.  Let’s make words have some meaning?  And if you do a preview and later run the adventure or the game, come back and do an actual review!  Compare your first thoughts to your eventual conclusions!


So here we are without a review culture to guide consumers (ooh, he said the “c” word!).  We have tastemakers and influencers but not reviewers.  Consumers - potential buyers of an adventure or a game - will see some guidance.  They are hungry for information to help them make a good decision about how to spend their money and time.  Without any reliable source of guidance to distinguish between products, we’re going to fall back on a) known producers of good reputation (be it writers or companies), b) impressive visuals (be it artwork or production values).  See where this leaves the independent producer?  In the mud.  If you’re a newish writer with a small budget, you will very quickly feel like you cannot compete.  And you can’t. 


Why can’t we talk like adults about products?  Why can’t we talk like adults about art?  There’s a gulf between “being courteous in criticism” and “avoiding all negativity”.  It’s nonsense.  We’ve gone beyond courtesy to cultishness.  You can’t criticize modern (postmodern?) works without being dogpiled.  (You can, of course, criticize things from before 2000 with impunity, and are encouraged to do so, because obviously they’re all Neanderthal crap.)  In OSR/indie circles, it’s perfectly acceptable to say “80% of what’s on DM’s Guild is hastily-produced shovelware that nobody will actually use at the table”. Dare I say it about the arthouse/indie OSR?  I suppose I do: “80% of what the OSR produces is farty shovelware that nobody will actually use at the table.”  Does that make it valueless?  Maybe, maybe not.  But we’re at the point where we’re afraid to express opinions and to disagree civilly.  It’s silly.  (Leave aside for the moment the idea that “proper” OSR mindset doesn’t really need anybody else’s product, as that’s a can of purple worms for a different time.)  Meanwhile, we have also somehow decided that we should only “review” products we like (because our time is valuable, and criticism is mean?), so we’re not learning anything from our (collective) mistakes because we aren’t talking about them.


While we’re at it: social media folks and YouTube types, please stop pretending that vagueposting about a problem you had with a product you refuse to name is helpful to the hobby.  It makes me want to call you and tell you there’s a small leak in your basement; no, I won’t tell you where.


We gave up criticizing “scene” products because it felt dirty.  It’s fine to yell at WotC, or TSR, or whoever.  Publicly criticizing another scenester, however, that’s backstabbing for triple damage, and completely verboten.  Unless you have a full-on beef, of course, in which case we’ll all gladly drink in the drama.  But actually talking about someone else’s work in a civil way?  Nope, can’t have it.  Not only do we have a layer of “you’re fine just as you are and shouldn’t be goaded into self-improvement”, but we have the scenester layer of “we’re all in this together and we don’t talk crap about each other”.  It’s nonsense, because it doesn’t improve the products, the people, the hobby, or the games.  It encourages everybody to try to be an unassailable auteur (“artiste”) and then get depressed when their macaroni-art high-concept adventure doesn’t sell five hundred copies.  You think rpgs are a community?  What functioning community sets its own members up for failure like that?  That isn’t a guild of artisans who teach one another, that’s a giant crab bucket.


I’ll come back to OSR marketing in another post and try to keep this one to the topic at hand - reviews are broken.


How do we fix it?  We reverse course.  We establish that it’s okay to talk about products and what you didn’t like about them.  What didn’t work for you, what didn’t work at the table.  We place the focus of the hobby back on the DOING - the actual playing of the games, and the discussion about what worked and didn’t.  We normalize accepting civil criticism civilly.


Play the games.  Talk about playing the games.




(Side-note: watch the interaction numbers on the post climb, since it's "Drama". And that's precisely the problem...)


13 comments:

  1. I agree reviews would be helpful if the reviewers had actually run the adventure. There is a recent trend for adventure innovation that mostly comes in the form of layout, but I’m less certain that trend is as innovative at the table as people believe. The problem with old gnarly reviewers is that they’re too busy looking for something new to them, whereas a regular old standard adventure (giant rats in the cellar) might be fun to someone with less experience. There is no perspective-taking.

    ReplyDelete
  2. I would like the review culture to be as you describe.

    Right now, I have to agree with your assessment of the present - I can muster the enthusiasm to write about the things I really loved either reading or taking to table but for anything else? Not worth the pain.

    ReplyDelete
  3. I have a couple of thoughts here. (1) It's a collective action problem. Until enough people are doing it, reviewing people's stuff critically in a hobby scene where many people participating are also making stuff burns social capital. We're navigating these weird parasocial relations with everyone and being harshly critical (when justified) complicates relations, especially when the thing being critiqued is a labor of love. So I think we would need to get past a tipping point or something of people doing real deal reviews.

    (2) There's a big obstacle to this kind of review culture: it's a lot harder to run something than to play a video game, or read a book, or whatever. I have two groups in a 5 year campaign going in my own world right now, so I can't run other rulesets, or most adventures. This standard would say to me: don't review things, even when you read them and have thoughts about them. Or at least don't call it a review.

    Maybe that's OK, but I suspect most people are like me. What you would need to get reviews in any quantity would be people who make their whole gaming activity running tons of different stuff. (But there's a paradox here, because that would make the reviews less interesting for me, because I don't like that style of play. I like long campaigns rather than one-shots.)

    ReplyDelete
  4. @Ben - definitely agree you'd need a tipping-point. Do you get feedback from people who are using your stuff?

    ReplyDelete
    Replies
    1. Not really no. Some reviews of the first two issues of Through Ultan's Door made good critical points that I tried to incorporate moving forward, especially Gabor Lux. Sometimes people start talking about my zines spontaneously (on a discord or whatever) and don't realize I can read what they're saying. And then I get actual feedback. (To be clear, I find reading feedback hard, so it's not like I'm really complaining, but it probably does make my stuff less good that I had to put my ear up to the door and listen to hear what people think could be better.)

      Delete
  5. @ Erik:

    All right, you made me just do a read-thru of every blog post I've made with the "review" tag, just to see if I've been doing this. Jeez...I have other things to do this morning!
    ; )

    Fortunately, I haven't been too bad: the vast majority of my "reviews" have been for films or TV shows (which is its own silliness...) or post about people who've reviewed MY works (with links). The remaining handful are, generally, reviews of Fantasy Heartbreakers (i.e. clones) explaining their differences from the games they emulate, or the occasional adventure that was sent to me requesting a review (a practice that I've since ceased...the last being in 2015). So...yay, me. My own thoughts on the practice of reviewing game products, I think, run largely parallel to yours:

    http://bxblackrazor.blogspot.com/2015/12/regarding-reviews-and-rants.html

    Personally, I don't buy much into the tribalism of the RPG community...that is, I don't feel like I have to defend (or protect) the indie guy/gal just because I happen to be kind of indie myself. This has rubbed some folks wrong in the past, but I try to be honest, and my criticism (which tends to be more negative) isn't intended to imply "stop writing/designing!" More like "work on improving this!"

    But, yeah...it's hard to be honest about a product that you haven't actually used at the table.

    Just how to rectify the situation is a bit of a puzzler. There is a LOT of product on the market, and unless you had a group of people dedicated to running games specifically for review purposes, it would be tough to get accurate play-test reviews. Amusing (or repetitive) as Bryce's reviews are, at least he's identifying adventures that aren't adventures but (rather) wastes of time and money. That's a valid service!

    Anyway.

    The times I've written (briefly) about a need for any sort of quality control I've heard shouts of "gatekeeping!" a term which appears to rouse the rabble into a hard-shell formation bent on protecting itself. To me, that's a sign that the desire for actual improvement is going to be an uphill slog. Yes, it's a giant crab bucket...but it's one that has proven lucrative for some folks, and they will continue to fight for that cash flow.

    It's a little depressing...but it's indicative of our times. Those of us who care just need to keep standing for quality and accountability and hope to make some positive impact.

    ReplyDelete
  6. "What I Liked/What I Didn't/How I'd Run It" is about my favorite format for TTRPG reviews. They're inherently idiosyncratic: imagine if each film projectionist had Strongly Held Opinions about how to operate a movie projector. There'd be no way to see the *same* movie as your friend 2 states over.

    ReplyDelete
  7. I've been asked before by people if I would review their work, so I show them examples of my reviews on my blog, but I've had people get upset when I point out issues with spelling or grammar (I'm also one of the Judges for the annual One Page Dungeon Contest and I can't stand consistent spelling/grammar errors; 1-2 is okay, but not preferable, but more than that, I'm docking the score). And, I also have an issue with whether the product I'm reviewing makes it clear who and what it's for - its intended purpose. I'll also comment on usability (how easy is it to find the info needed to use this, which does often related to layout). And yet I will get folks who seem surprised when I mention these types of things in my reviews.

    I do think there's an expectation in this community that if someone asks, "Will you review my product?" what they are really asking is, "Will you give my product a positive review?"

    I have begun considering that I might shift to only reviewing things that I actually like, for several reasons. That's the stance that Professor Dungeon Master from Dungeoncraft uses, and I think there's something to that. His reviews center around "what from this can you actually use in a game, and how could you use it to make your games better?" That's a good approach, I think and one that I'll probably concentrate on moving forward.

    ReplyDelete
  8. Potential issues:

    A standard 30-room dungeon could take anywhere from 1 to 10 sessions to run, depending on the group, system, etc. As a one-shot, that's not a huge time investment. As part of an ongoing game, that's potentially weeks of play, just for a review. If it's a bad module, everyone is unhappy for very limited benefit.

    It's easy enough to define what it means to run a dungeon/adventure, but what about a setting guide/bestiary/system/set of tables? Do I need to run every monster? Run one-shot or a long campaign? Several campaigns with different groups?

    If I read a module, find a mechanic that doesn't seem to work, run a quick test and confirm that it doesn't work as intended, do I need to test it in front of a group to confirm that it's broken/misphrased/untested? If I run the module (as suggested) to provide a full review, should I modify/fix the mechanic (to improve the group's play experience) or run it as written (to ensure the group's feedback matches the module as written as closely as possible)?

    I wouldn't feel confident writing content or running games if I didn't have some idea of what makes a good game without testing every bit of content. Testing might refine an idea, or reject the occasional blockheaded blunder, but you can't GM by the neural network method of "test everything and see what sticks." There has to be some degree of insight before the game starts.

    Theory:
    I think RPG books are sort of like mise-en-place. Meal prep. The dining experience (from cooking to serving to conversation at dinner) is the game.

    It's possible to say "these onions are chopped strangely" or "the recipe calls for bean sprouts but no bean sprouts were provided" or "this lettuce has gone brown and supports a lively herd of caterpillars", and for those to be accurate judgements, without needing to cook or serve the meal. It's tricky to tell if a dish will be delicious or overseasoned without cooking it, but you can still make some judgements and evaluations about the potential meal based on the prep alone.

    This analogy is very silly.

    ReplyDelete
  9. Not that it'll help with OSR stuff, but Seth Skorkowsky makes a point of always having played/run anything he reviews. Perhaps a look at his work might be inspirational for how to approach reviewing things in a reasonably objective manner. His Youtube channel's here if you're unfamiliar with it:

    https://www.youtube.com/channel/UCQs8-UJ7IHsrzhQ-OQOYBmg

    I also note that Questing Beast just released his first system-agnostic, OSR-friendly adventure module, so maybe you'd like to review it yourself and show us an example of what you'd like to see. I've watched his Ultan's Door reviews (no quotes required IMO) and they certainly went well beyond paper quality and art.

    ReplyDelete
  10. Thanks for the interaction, everybody - I appreciate it.

    @Dirk - you're not the first person to recommend Seth, I'll check him out. Thanks! And I have that QB adventure, but since I've only flipped through it, I will withhold any substantive commentary on it until I've run it.

    @Skerples - I appreciate a food metaphor that goes on too long as much as anybody (maybe more).

    @JB - Improvement is always an uphill slog!

    ReplyDelete
  11. Here's what I hope is a useful contribution.

    https://grumpywizard.home.blog/2021/03/10/review-the-punchline/

    ReplyDelete