Monday, March 8, 2021

What's Broken: Reviews

 On a regular cycle, someone complains about the death of the OSR (whether it was 2013 or 2017 or yesterday or tomorrow), and inevitably somebody chimes in talking about how things got “too commercial” and everybody in the “scene” wanted to make a buck and crank out a product.  That’s worth talking about, but there are some other issues that dovetail with it that I think I need to discuss first so we get a complete picture of where I’m headed with all this.



One core issues I see - and I’ve occasionally complained about it on social media, I suppose - is our lack of a review culture.  We don’t have meaningful reviews to consult and compare.  We have shilling and we have tear-down rants, and very little in-between.


There are two large classes of OSR “reviews”:


  • THIS IS BAD AND I HATE IT

    • “This is not written the way I would write it” (fair enough, but diminishing returns as to usefulness of this sort of review)

    • “This classic product sucks/can’t-be-run” (often with a side dish of “I’m much cleverer than these 70s-80s people”, which is distasteful even when true)


  • THIS IS AWESOME AND I LOVE IT

    • “Everything this guy/company does is equally great”  (usually not true, and how could it be?)

    • “This looks really high-quality doesn’t it”  (so?)

    • “This is super-innovative, subversive, artsy and I love it.  Also, my friend made it.  No, I haven’t run it LOL who actually plays games?”

    • “I used parts of this, twisted to meet my needs under a different ruleset, and I changed most of it” (interesting reading, and a standard byproduct of the way we hobby, but perhaps less useful to some readers)


With most of both of these written without ever having run the product.  Imagine how different our discourse might be if we had the expectation that once you ran an adventure, you actually talked about how it went.  Instead, we’re stuck in a rut where we have a “balance” between rants and stroking.  Bryce Lynch writes reviews which are sometimes amusing to read, but have become so repetitive as to be much less useful than they used to seem (whether this is Bryce’s fault or that of the endless stream of mediocre adventures he dares to read is a separate question).  Fear of a Black Dragon produces interesting reviews much of the time, although often they take a larger work and bang it into a rules-light shape.  Questing Beast usually does “reviews” that are more about the paper quality and the art than about using the material at the table.  Some old-school bloggers go in-depth into “what’s wrong” with a TSR module - and maybe even attempt to provide solutions - but never bother to run the adventure, even WITH the proposed fixes.  


Can’t we have reviews by people who ran the product?  Answer: perhaps not if we want reviews to appear during the main sales push cycle for a product.  Yet if everything’s available forever, I’d happily take a robust long-tail review culture over what we have now.


Here’s a call to action: if you haven’t actually used the product, don’t call it a review.  Have some dignity.  Call it a “first look”, a “preview”, an “impressions”, whatever.  Let’s make words have some meaning?  And if you do a preview and later run the adventure or the game, come back and do an actual review!  Compare your first thoughts to your eventual conclusions!


So here we are without a review culture to guide consumers (ooh, he said the “c” word!).  We have tastemakers and influencers but not reviewers.  Consumers - potential buyers of an adventure or a game - will see some guidance.  They are hungry for information to help them make a good decision about how to spend their money and time.  Without any reliable source of guidance to distinguish between products, we’re going to fall back on a) known producers of good reputation (be it writers or companies), b) impressive visuals (be it artwork or production values).  See where this leaves the independent producer?  In the mud.  If you’re a newish writer with a small budget, you will very quickly feel like you cannot compete.  And you can’t. 


Why can’t we talk like adults about products?  Why can’t we talk like adults about art?  There’s a gulf between “being courteous in criticism” and “avoiding all negativity”.  It’s nonsense.  We’ve gone beyond courtesy to cultishness.  You can’t criticize modern (postmodern?) works without being dogpiled.  (You can, of course, criticize things from before 2000 with impunity, and are encouraged to do so, because obviously they’re all Neanderthal crap.)  In OSR/indie circles, it’s perfectly acceptable to say “80% of what’s on DM’s Guild is hastily-produced shovelware that nobody will actually use at the table”. Dare I say it about the arthouse/indie OSR?  I suppose I do: “80% of what the OSR produces is farty shovelware that nobody will actually use at the table.”  Does that make it valueless?  Maybe, maybe not.  But we’re at the point where we’re afraid to express opinions and to disagree civilly.  It’s silly.  (Leave aside for the moment the idea that “proper” OSR mindset doesn’t really need anybody else’s product, as that’s a can of purple worms for a different time.)  Meanwhile, we have also somehow decided that we should only “review” products we like (because our time is valuable, and criticism is mean?), so we’re not learning anything from our (collective) mistakes because we aren’t talking about them.


While we’re at it: social media folks and YouTube types, please stop pretending that vagueposting about a problem you had with a product you refuse to name is helpful to the hobby.  It makes me want to call you and tell you there’s a small leak in your basement; no, I won’t tell you where.


We gave up criticizing “scene” products because it felt dirty.  It’s fine to yell at WotC, or TSR, or whoever.  Publicly criticizing another scenester, however, that’s backstabbing for triple damage, and completely verboten.  Unless you have a full-on beef, of course, in which case we’ll all gladly drink in the drama.  But actually talking about someone else’s work in a civil way?  Nope, can’t have it.  Not only do we have a layer of “you’re fine just as you are and shouldn’t be goaded into self-improvement”, but we have the scenester layer of “we’re all in this together and we don’t talk crap about each other”.  It’s nonsense, because it doesn’t improve the products, the people, the hobby, or the games.  It encourages everybody to try to be an unassailable auteur (“artiste”) and then get depressed when their macaroni-art high-concept adventure doesn’t sell five hundred copies.  You think rpgs are a community?  What functioning community sets its own members up for failure like that?  That isn’t a guild of artisans who teach one another, that’s a giant crab bucket.


I’ll come back to OSR marketing in another post and try to keep this one to the topic at hand - reviews are broken.


How do we fix it?  We reverse course.  We establish that it’s okay to talk about products and what you didn’t like about them.  What didn’t work for you, what didn’t work at the table.  We place the focus of the hobby back on the DOING - the actual playing of the games, and the discussion about what worked and didn’t.  We normalize accepting civil criticism civilly.


Play the games.  Talk about playing the games.




(Side-note: watch the interaction numbers on the post climb, since it's "Drama". And that's precisely the problem...)


27 comments:

  1. I agree reviews would be helpful if the reviewers had actually run the adventure. There is a recent trend for adventure innovation that mostly comes in the form of layout, but I’m less certain that trend is as innovative at the table as people believe. The problem with old gnarly reviewers is that they’re too busy looking for something new to them, whereas a regular old standard adventure (giant rats in the cellar) might be fun to someone with less experience. There is no perspective-taking.

    ReplyDelete
  2. I would like the review culture to be as you describe.

    Right now, I have to agree with your assessment of the present - I can muster the enthusiasm to write about the things I really loved either reading or taking to table but for anything else? Not worth the pain.

    ReplyDelete
  3. I have a couple of thoughts here. (1) It's a collective action problem. Until enough people are doing it, reviewing people's stuff critically in a hobby scene where many people participating are also making stuff burns social capital. We're navigating these weird parasocial relations with everyone and being harshly critical (when justified) complicates relations, especially when the thing being critiqued is a labor of love. So I think we would need to get past a tipping point or something of people doing real deal reviews.

    (2) There's a big obstacle to this kind of review culture: it's a lot harder to run something than to play a video game, or read a book, or whatever. I have two groups in a 5 year campaign going in my own world right now, so I can't run other rulesets, or most adventures. This standard would say to me: don't review things, even when you read them and have thoughts about them. Or at least don't call it a review.

    Maybe that's OK, but I suspect most people are like me. What you would need to get reviews in any quantity would be people who make their whole gaming activity running tons of different stuff. (But there's a paradox here, because that would make the reviews less interesting for me, because I don't like that style of play. I like long campaigns rather than one-shots.)

    ReplyDelete
  4. @Ben - definitely agree you'd need a tipping-point. Do you get feedback from people who are using your stuff?

    ReplyDelete
    Replies
    1. Not really no. Some reviews of the first two issues of Through Ultan's Door made good critical points that I tried to incorporate moving forward, especially Gabor Lux. Sometimes people start talking about my zines spontaneously (on a discord or whatever) and don't realize I can read what they're saying. And then I get actual feedback. (To be clear, I find reading feedback hard, so it's not like I'm really complaining, but it probably does make my stuff less good that I had to put my ear up to the door and listen to hear what people think could be better.)

      Delete
  5. @ Erik:

    All right, you made me just do a read-thru of every blog post I've made with the "review" tag, just to see if I've been doing this. Jeez...I have other things to do this morning!
    ; )

    Fortunately, I haven't been too bad: the vast majority of my "reviews" have been for films or TV shows (which is its own silliness...) or post about people who've reviewed MY works (with links). The remaining handful are, generally, reviews of Fantasy Heartbreakers (i.e. clones) explaining their differences from the games they emulate, or the occasional adventure that was sent to me requesting a review (a practice that I've since ceased...the last being in 2015). So...yay, me. My own thoughts on the practice of reviewing game products, I think, run largely parallel to yours:

    http://bxblackrazor.blogspot.com/2015/12/regarding-reviews-and-rants.html

    Personally, I don't buy much into the tribalism of the RPG community...that is, I don't feel like I have to defend (or protect) the indie guy/gal just because I happen to be kind of indie myself. This has rubbed some folks wrong in the past, but I try to be honest, and my criticism (which tends to be more negative) isn't intended to imply "stop writing/designing!" More like "work on improving this!"

    But, yeah...it's hard to be honest about a product that you haven't actually used at the table.

    Just how to rectify the situation is a bit of a puzzler. There is a LOT of product on the market, and unless you had a group of people dedicated to running games specifically for review purposes, it would be tough to get accurate play-test reviews. Amusing (or repetitive) as Bryce's reviews are, at least he's identifying adventures that aren't adventures but (rather) wastes of time and money. That's a valid service!

    Anyway.

    The times I've written (briefly) about a need for any sort of quality control I've heard shouts of "gatekeeping!" a term which appears to rouse the rabble into a hard-shell formation bent on protecting itself. To me, that's a sign that the desire for actual improvement is going to be an uphill slog. Yes, it's a giant crab bucket...but it's one that has proven lucrative for some folks, and they will continue to fight for that cash flow.

    It's a little depressing...but it's indicative of our times. Those of us who care just need to keep standing for quality and accountability and hope to make some positive impact.

    ReplyDelete
  6. "What I Liked/What I Didn't/How I'd Run It" is about my favorite format for TTRPG reviews. They're inherently idiosyncratic: imagine if each film projectionist had Strongly Held Opinions about how to operate a movie projector. There'd be no way to see the *same* movie as your friend 2 states over.

    ReplyDelete
  7. I've been asked before by people if I would review their work, so I show them examples of my reviews on my blog, but I've had people get upset when I point out issues with spelling or grammar (I'm also one of the Judges for the annual One Page Dungeon Contest and I can't stand consistent spelling/grammar errors; 1-2 is okay, but not preferable, but more than that, I'm docking the score). And, I also have an issue with whether the product I'm reviewing makes it clear who and what it's for - its intended purpose. I'll also comment on usability (how easy is it to find the info needed to use this, which does often related to layout). And yet I will get folks who seem surprised when I mention these types of things in my reviews.

    I do think there's an expectation in this community that if someone asks, "Will you review my product?" what they are really asking is, "Will you give my product a positive review?"

    I have begun considering that I might shift to only reviewing things that I actually like, for several reasons. That's the stance that Professor Dungeon Master from Dungeoncraft uses, and I think there's something to that. His reviews center around "what from this can you actually use in a game, and how could you use it to make your games better?" That's a good approach, I think and one that I'll probably concentrate on moving forward.

    ReplyDelete
  8. Potential issues:

    A standard 30-room dungeon could take anywhere from 1 to 10 sessions to run, depending on the group, system, etc. As a one-shot, that's not a huge time investment. As part of an ongoing game, that's potentially weeks of play, just for a review. If it's a bad module, everyone is unhappy for very limited benefit.

    It's easy enough to define what it means to run a dungeon/adventure, but what about a setting guide/bestiary/system/set of tables? Do I need to run every monster? Run one-shot or a long campaign? Several campaigns with different groups?

    If I read a module, find a mechanic that doesn't seem to work, run a quick test and confirm that it doesn't work as intended, do I need to test it in front of a group to confirm that it's broken/misphrased/untested? If I run the module (as suggested) to provide a full review, should I modify/fix the mechanic (to improve the group's play experience) or run it as written (to ensure the group's feedback matches the module as written as closely as possible)?

    I wouldn't feel confident writing content or running games if I didn't have some idea of what makes a good game without testing every bit of content. Testing might refine an idea, or reject the occasional blockheaded blunder, but you can't GM by the neural network method of "test everything and see what sticks." There has to be some degree of insight before the game starts.

    Theory:
    I think RPG books are sort of like mise-en-place. Meal prep. The dining experience (from cooking to serving to conversation at dinner) is the game.

    It's possible to say "these onions are chopped strangely" or "the recipe calls for bean sprouts but no bean sprouts were provided" or "this lettuce has gone brown and supports a lively herd of caterpillars", and for those to be accurate judgements, without needing to cook or serve the meal. It's tricky to tell if a dish will be delicious or overseasoned without cooking it, but you can still make some judgements and evaluations about the potential meal based on the prep alone.

    This analogy is very silly.

    ReplyDelete
  9. Not that it'll help with OSR stuff, but Seth Skorkowsky makes a point of always having played/run anything he reviews. Perhaps a look at his work might be inspirational for how to approach reviewing things in a reasonably objective manner. His Youtube channel's here if you're unfamiliar with it:

    https://www.youtube.com/channel/UCQs8-UJ7IHsrzhQ-OQOYBmg

    I also note that Questing Beast just released his first system-agnostic, OSR-friendly adventure module, so maybe you'd like to review it yourself and show us an example of what you'd like to see. I've watched his Ultan's Door reviews (no quotes required IMO) and they certainly went well beyond paper quality and art.

    ReplyDelete
  10. Thanks for the interaction, everybody - I appreciate it.

    @Dirk - you're not the first person to recommend Seth, I'll check him out. Thanks! And I have that QB adventure, but since I've only flipped through it, I will withhold any substantive commentary on it until I've run it.

    @Skerples - I appreciate a food metaphor that goes on too long as much as anybody (maybe more).

    @JB - Improvement is always an uphill slog!

    ReplyDelete
  11. Here's what I hope is a useful contribution.

    https://grumpywizard.home.blog/2021/03/10/review-the-punchline/

    ReplyDelete

  12. As we are aware of the fact already that commercial inspection is very different from residential inspection hence the expertise needed for this to be done are different as well. Commercial inspector is someone who has the necessary experience needed for a commercial inspection which needs certifications and education required by the state along with a portfolio that states that they have done it before and the customers were happy and satisfied with the outcome of the inspection. As neglected as this perspective can be because there is more work when you are just a home inspector, being a commercial building inspector can be pretty rewarding as well because commercial buildings obviously cover more area hence the money will be good and as commercial building inspections are covered by companies, everything will be done more professionally along with the less hassle of having to involve in negotiations. All in all, commercial inspection can be pretty rewarding if a real estate inspector gets a good grip over it.

    ReplyDelete
  13. While there are a ton of residential inspectors present around us always luring us to get a home inspection done from them and many home inspection companies are offering numerous perks if we get a home inspection done from them, when it comes to a commercial inspection, the list narrows down to a limited number and the reason behind this is that commercial property inspection is something on a bigger level and it requires a lot of expertise in order to get it right. When we search with words like commercial property inspection near me while residing in Atlanta, we may get a handsome amount of commercial inspectors on the search results but we should know what to look for before taking any decision. Champia Real Estate Inspections is a company offering expert services for people who are looking for a skilled and experienced commercial real estate inspector to help them in their time of need. They have been working since 1987 and have a number of positive reviews which is a guarantee itself that you will not be disappointed when to get their services.



    ReplyDelete
  14. Commercial property inspection is a term which is used when the help of a real estate inspector is needed on a bigger level. It is not just about a house anymore. In order to get done with this process, experience and skill is needed because we are never sure what will be in the box for us because in a place like Atlanta, commercial properties can be of any size and it is not possible for a regular home inspector to bluff on this level that he can satisfy the clients completely. Commercial property inspectors are experts who know exactly what they are dealing with. They are aware that anything can be present on the commercial property and it can be of any size so they need to be prepared prior to the inspection day and they should know how they are going to satisfy the client who is looking for their help regarding their commercial property.

    ReplyDelete
  15. In an industry like Home Inspection Answering Calls Solution is very important as you get your job through calls so if you miss the calls you are getting as inquiries, you will eventually lose the sales. Home inspection services hold the essence of a good impression by delivering 100% customer satisfaction which means that at the end of the day, if your customers are not happy with your services, this will impact your business badly but a very important point that is missed that a person will only book an inspection if he is convinced on call about what he will be provided with once he does it. If a person calls a company, worried about the state of his house and call is not picked, he will definitely call the next best option to fix this problem. And even if the call is attended and he is not satisfied with what he is being told, he will not proceed further and book an inspection. This is where people need to understand the importance that Call Center Solution for Real Estate holds. If a person is satisfied, he will definitely give you the job and spread positive word of mouth about the services he got from the moment he called the company till the moment the inspection was done.

    ReplyDelete
  16. While working as a home inspection company, the actual job for them is to complete successful home inspections in which the customer is satisfied with their services at the end of each inspection and is happy to leave a positive review but what happens when the matter is not closed to book a home inspection? What if you are busy delivering your best services and while doing so, you miss a call that could actually be a successful sale for you? This is why we recommend you to get an Outbound Calling Service Home Inspection requires your complete attention and that is why ICC is here for you to take your calls and satisfy the customers with adequate answers to their queries. Homeowners, while looking for an impeccable home inspection schedule appointments in a rush because they are concerned about the state of their house. This is where we come in. We make sure to attend their calls in time, satisfy them about the services they will be given and prepare a proper tentative schedule for you to just go and perform your job without worrying about the hassle of picking the calls of potential clients.

    ReplyDelete
  17. A popular concept is present in the market which is termed as the cost of a missed call which is not understood by all. People are not aware of the importance of an inquiry call as there are many coming every day and when you are working in a home inspection company, you are often busy with the actual task which is a successful home inspection but how will you get a job of performing a home inspection if you keep missing calls? In the industry of Home Inspection Lead Management is important because 1 out of 5 calls can be potential and losing a potential call will end up in you losing a sale which will affect the business. When we talk about Customer Service Home Inspection industry needs it the most because when people call a home inspection company, they are often worried about the condition of their house or what affect it will be having on the health of the residents of their house so in order to make sure that a customer is satisfied with the response he gets when he calls a company for help. This is where Inspector Call Center comes for your rescue by saving your business from losing potential sales and leaving your callers content.

    ReplyDelete
  18. Suppose a person is working solely as an owner of a home inspection company and he does not need to hire more people as he thinks that he is doing his job perfectly and does not want to add more people to it because they may ruin it for him by providing compromised services but as he is always on the go to provide his best services, he often tends to miss calls or cannot respond to them because he is busy all the time. Then he comes across a solution which means that all his calls will be answered by someone else who will make sure to do this job perfectly giving him more time to complete his inspections peacefully and more jobs are waiting for him as all of his inquiry calls are getting attended. When provided with a proper Customer Care solution home inspection became a stress free job for him as he knew that he only has to focus on one thing which he does the best. Believe it or not but once he became aware about the benefits of Inspector Service Group his life became tension free and his sales boosted up instantly because no potential calls were being missed.

    ReplyDelete
  19. In Atlanta real estate inspection is considered an important factor before proceeding with that property, whether its a workplace or your home. The fact that wherever you are going to spend the maximum of your time should be healthy for you. While we take care of the cleanliness of the space we are working in, taking care of the air quality around us is important as well. One of the most dangerous toxin that is present in the air around us is radon. Radon gas is present in the air around us but when it gets trapped around us inside our property where we breath, it can get stuck inside our system and its prolonged inhalation is the leading cause of lung cancer in non smokers. Champia real estate inspections offer bestRadon testing in Atlanta as they have the necessary equipment and experienced professional with relevant expertise and a huge number of positive reviews so one can completely rely on them regarding your safety and satisfying results.

    ReplyDelete
  20. In Atlanta, Real Estate Inspections are an important part to be included in your property maintenance routine because as much as getting your painted or getting apparent repairs done is important, a home inspection is important in order to know if your house is safe for you and your family to live or not. There are many factors that are hazardous for your family such as high levels of radon inside your home, mold problems, water leaks, termites, air pollution and other factors that pose a threat to your health. A known name for Home Inspection Atlanta has is Champia Real Estate Inspection which deals in home inspection as well as commercial inspections. They were founded in 1987 and have a record of over 50,000 completed inspections and a completion time of 24 hours which is pretty quick. Not to forget their multiple warranties on home inspection which makes it even more reliable.

    ReplyDelete
  21. When it comes to home inspection Atlanta ga has the best home inspection company who will deal with all your problems relating to the health of your house. One of the very common problems that are present in almost every house either known or unknown is the presence of mold inside the house. Mold is present around us everywhere so it is nearly impossible to eradicate them completely unless it is some heavy mechanical equipment. They are present in the air in the form of spores and spread on moist environments. It is a good new and bad one too because if you have moist surfaces at your home, there is a good chance that mold is present around you too and it can cause many harms to your health. It is also a reason to worsen present medical conditions like allergies and respiratory diseases. Champia offers the best Mold testing atlanta has. They have necessary expertise and equipment to deal with this problem and help you get rid of it.

    ReplyDelete
  22. When it comes to Mold Testing Champia real estate inspections have the best services to offer. As they have the best home inspectors in Atlanta who have the necessary expertise to perform a successful home inspection and adequate experience. They have completed over 50000 inspections and a great number of positive reviews as a proof that you can rely on them completely when it comes to the healthy environment of your home. They provide a number of warranties as well for residential inspection so you can be relaxed about the outcome of the inspection. For residential inspection, they give 24 hours ultimatum and schedule and complete the appointments and reports at the exact time. The home inspectors at Champia are very adaptive and care about your comfort. They love it when you are an active part of the home inspection because in this way you can get to know more about the details and they can give you more tips about the maintenance about how to take care of your house after the inspection process.

    ReplyDelete
  23. When it comes to Home Inspections Atlanta Georgia has the best inspection company named as Champia Real Estate Inspections because they were founded in 1987 and since then they have been serving with excellence. They have over 50000 completed inspections in their portfolio and have diverse expertise. This real estate inspection company deals with residential as well as commercial home inspections and have a delivery time of reports in 24 hours for residential inspections and around four days for commercial inspections. Their quick and efficient service plus their customer satisfaction rate is what makes their best real estate inspection and other states of Atlanta has to offer. Among many others, the three key qualities that make them prominent are their experience, certifications and their affordability. They come with additional warranties for residential inspections only so that the homeowner is comfortable with the services and the person that they are letting inside their house.

    ReplyDelete
  24. There are two very important things that come to our mind when we think about a home inspection:
     Is it safe to let a stranger inside our house?
     Is spending money on this worth it?
    We have the answers for you. While residing in Atlanta, and along with many options to choose from, Champia Real Estate is the best choice for you. While spending on getting a home inspection from them, it is affirmative that in whole Atlanta home inspection prices are very affordable as compared to the quality of services they are providing. They make sure that the money you are spending is worth the hustle. They involve you in the process to give you proper insights and maintenance tips. home inspectors in atlanta sent by Champia are certified professionals and they come with insurance plus different sorts of residential warranties so you can totally rely on them when it comes to these two terms which can make you worry about the process.

    ReplyDelete
  25. Atlanta home inspection is a booming industry because of the many cases found inside the homes which were proven to affect the health of the residents and the structure of the house. House inspection companies help out the residents in finding out what is wrong with the place they are residing or even if they are buying that property. Champia Real Estate inspections have the best home inspectors Atlanta has to offer because of their vast experience and their expertise in this field. Champia Real Estate Inspections has been working in Atlanta since 1987 and have a great number of positive reviews in their pocket to back their services. Their home inspectors are friendly and adaptable hence they make it easy for the customer to address their issues to them in order to understand the situation completely and take the necessary steps needed. To make it more comforting, they back their services with a number of warranties so that customers can rely on them completely.

    ReplyDelete