MSP Open Judging - What you were afraid to ask

[Heisler]

This wall of text post is the result of a private conversation I have had with someone here on the forums. I have edited it a bit and removed names and things of that nature. Hopefully it may answer some of those questions people can be afraid to ask. I'm happy to have conversations like this and I thought that this might spur some more questions that people are uncomfortable asking,

 

It starts off like this

Let me ask you honestly, do you think that some things are judged more harshly for the genre they are painted in such as some may like say historical better than fantasy?

Also, do you feel there are ever biases for people who've gotten awards in the past or are "well known" or "friends of" the judges?

Reason I ask is because I saw some and looked very closely at a few figures and things that I got "dinged" for seemed to be overlooked on figures which got higher marks. 

I'm not calling anyone out or trying to start trouble, but I was a bit put off by how varying the standards were and I talked to a few friends who had certain judges that'd mark at say a Bronze and one would mark at a Gold citing that the bronze didn't like the technique, but the gold uses it and loves it. I felt some were inconsistent. 

Really, I just want to know how to "level up" and be able to achieve higher standards with how varying the judging is. It's a bit confusing at this point.

 

Those are hard questions for early in the morning, quite frankly they are hard questions period because we are dealing with people in a subjective environment. I'll do my best to explain it. 

 

As you know a team consists of three judges who after deciding which piece to score (if there are multiple entries in a category) then assign a score of 0-4. Michael and I do instruct the judges (and we are usually on teams as well) spending about 15 minutes refreshing everyone on what we are trying to accomplish. In general judges on a team don't consult with each other on what score they are going to give, they will consult with each other if they are struggling on how to score something. In general Michael and I both tell them to start with silver score (a 3, we also try to avoid giving 0s at all, except in the youth division) and then move up or down from that point.

 

Do judges have a bias against different genres? While I don't believe that judges have particular biases between genres there are certainly areas where judges are stronger or weaker. For instance, we have judges that we won't put on a team that will cover vehicles and ordnance because its weak area for them. However, they are excellent when judging in the painters category. I think it has more to do with how comfortable a judge is within a category rather than a bias towards a particular genre.

 

Do judges have a bias for past award winners, or friends? This is a really tough question and its one of the reasons there are three judges on a team and not a single person doing it. There is a kind of a yes and no answer to this question. As judges we do expect a higher quality of work from a past gold medalists, Sophie winners, best of show winners or even instructors. In general they probably get judged a bit more harshly because of their past work, we know what they are capable of. Do we maybe allow some of them to dial it in anyway, quite possibly, again it depends on the judging team and quite frankly when someone like Jen Haley or Jess Rich dials it in they are still head and shoulders above the rest of us.

 

Friends are a bit of different story, in general we tell judges that they need to bow out of judging a piece that they have a vested interest in because it was done by a friend or because they consulted a significant amount on it (I was pulled into this myself this year when they were judging the Dark Sword awards). We always have alternate judges available to fill in when something like this happens and it is not an uncommon occurrence. Unfortunately, it is not always clear when something like this has happened because you have to see the physical judging sheet to see if a different set of initials are in the scoring box for that piece. [Edit: Judges don't score their own pieces either, this is another place where the alternate will step in.]

 

Are judges inconsistent? [this was my initial answer "To a certain extent yes". As I reread this I have changed my mind]. Judges tend to be very consistent with how they score things (I have been tracking judges' scores for years now), what is inconsistent is that no two judges view entries in quite the same light, we are human after all. We make mistakes or we are grumpy because we have been on our feet all day teaching and that can be unconsciously reflected in how an entry gets scored. While Michael and I try to catch the obvious ones like a judge scoring a 4 and another judge on the team scoring the same entry a 1 scores like a 1, 2, 3 (which adds up to a  bronze) are not uncommon. We are dealing with someone's opinion on what they are looking at. Remember, in general, the judges don't discuss how they are going to score a piece with each other. If there are multiple entries the discussion is more along the lines of "I can score this piece higher than that one".

 

We try very hard to encourage discussions in positive terms not negative terms. That doesn't mean it doesn't happen and many of our judges have a background in trophy judging where the first thing you are looking for are flaws. We want our judges to look at the good things first.

 

I can't speak to the last question because I would have to see the circumstance. Most judges don't put down comments on the scoring sheets (we can, but often time gets away from us). For your particular example I would have to know what entry is being talked about and what category it is in and who the three judges were. Also what was the third score?

 

I do track the judges' scores and we try and make sure teams the teams are balanced. Some judges in general are tougher than others. I don't want two "tough' judges on the same team nor do I want to two "easy" judges on the same team either. In general, I think that the results are for more consistent than most people realize but unless you are right there doing the scoring and involved with the conversations it is hard to see that. And yes, we have had individuals that are not particularly good at judging despite how good a painter and instructor they may be. For that one year, they are going to have an influence on the entries they score in the competition. The judges, like the entrants, are human and have their own inherent biases and opinions. By using three judges we hope to achieve a balance and get realistic scores for every piece that was entered.

 

With our need to get more judges into the pool we did run a bit of an experiment this year. In vehicles and ordnance, I judged the category with two newbies to the ReaperCon competition. However, they are both experienced judges in that category at other shows. I was pleased to find that our scores were almost in lock step with each other with a few 1 point variations which I would expect to find. Will they do as well as we introduce them into the painter, open, and diorama categories? I don’t know, but I would be comfortable using them as dedicated vehicle and ordnance judges now. This lets me use a third judge that is not familiar with the category get experience under their tutelage.

 

I also have people asking how they can become judges. A hard question and not one that you asked. We like our judges to have some experience as instructors first. The first thing we need from a judge is the ability to communicate with people and be familiar explaining techniques and able to give good critiques. The easiest way for us to do that is make sure they are good instructors first. It doesn't always mean that they are going to be good judges but it gives us a starting point. The first year a new judge serve as an alternate. They are typically assigned to a single team for that year and step in when a primary judge needs to step out. That way they get the advantage of being involved in the discussions and can get in and score a few things themselves. The year after that we try and work them in as a primary judge on a team, that is usually a make or break year for us to figure out if they are going to be a good judge or not.

 

I'm not sure that I have answered your questions very well. It is hard to sit down, based on the judging, and figure out what you need to do to "level up". We emphasis the feedback portion of the competition but sometimes that can cause confusion when you are getting hit with multiple opinions. I think the best critiques start out with the judging asking you what you think you did really well and what you didn't like. Often you know what needs to be done you just haven't acknowledged it. 

 

Thanks for taking the time to write out such a great response, Kris!

I really do appreciate it. I know it's very subjective, and all judges are human (well some may machines with how well they paint!). 

Really I guess it comes down to conflicting feedback from judge to judge for me. Some may love your blending and use of colors and give high marks, others may think it's crap and give you low marks. I was just curious if there are "standards" they should follow to try to keep personal feelings on techniques used out of the equation. A good example of this is drybrushing. I know a few judges who quite literally sneer at the usage and feel it's something that shouldn't ever be used and others who respect it as a tool in the toolbox to be strategically used if done well.

 

 

There is not a set of standards. It would literally be impossible to set something like that unless we selected a single artist to be our standard, so we have to lean on experience and skill.  To a certain extent you are always going get different opinions from the judges and that's were their own bias towards something is going to get in. It doesn't mean that they necessarily dinged you for that for scoring, unless they were quite specific about it.

 

We do instruct the judges that all techniques are valid, that they are judging the execution of that technique even if it is a technique that they don't use or don't think is effective. If you executed drybrushing correctly then you shouldn't get dinged for it even if the judge doesn't like that technique. 

 

Oh, no problem, it's mostly curiosity on my part at this point after talking to a few other friends and hearing their scores/feedbacks. 
 

Again, this wasn't about my particular judges for painters category. I got 2 bronze and 1 silver scoring so it was pretty consistent. I also got feedback from others like Jen Haley which was very valuable. 

And I'm not trying to cause trouble or stress, really want to be able to understand the system better so I can do better and know what people are looking for as they judge pieces as someday I would like to get golds and Sophies. I know that my biggest critiques are about contrast and I get it, but I personally like more subtle things like the Europeans are doing. Things with textures and grittiness. But I also know I should conform a bit because right now in painting in the US it seems to be all about the contrasts and smooth blends for judges. 

Again, thanks for your time in talking about this and I really do appreciate the thought you put into your responses.

 

Many of our judges have been exposed to the European styles and have taken classes with the European painters. They should be able to appreciate that style as well. Just keep in mind that in some ways the Europeans take contrast, especially between light and dark even more extremely than we do.

 

Really there are a heck of a lot of styles out there now. It just feels to me like the biggest thing people look for in these contests is contrast and always upping values to the extremes. I get it, these are tiny pieces and the lighting isn't the best to judge by so subtly will easily get missed in competition when you are looking over hundreds of entries and you're tired and have certain things each judge looks for. 

I always welcome conversations on how to get better with painting and composition and techniques so a full forum conversation could be pretty good for everyone. 

I was a bit disappointed in myself for only getting bronze in painters as I felt that I leveled up on my pieces since last year when I also got bronze. I got a bunch of feedback from a lot of people on the forum and other places on my techniques. Really, I do just want to get better and am open to all advice and comments in order to do so. Taking classes and getting personal feedback are great ways to do this and part of the reason I love ReaperCon so much.

 

The leveling up in medals is really hard. One of the things that you have to remember is that while the scoring sounds simple:

0-1 - no award

2-4 - Certificate

5-7 - Bronze

8-10 - Silver

11-12 - Gold

the difference between all those scores is much bigger than it appears. The difference between scoring a 7 (high bronze) and an 8 (low silver) from the judging point of view is significant. I didn't do a lot of critiques this year since I only judged one category, but the single biggest thing I saw this time around was thick paint. Keep it thin!

 

While many of us are restricted to asking for advice and help online, it is hard to really to really get and give good feedback. Things I see in a picture may not be the result of your paintwork but more the result of your lighting. What looks great in person may look awful in a photograph and vice versa. Ultimately there is no magic bullet, you need to paint, paint and paint some more to get better. Which means that most people aren't painting enough to get from one level to another and if you take long breaks between painting you forget things and skills deteriorate. Its practice, practice, practice! Also if you are consistently painting for the table top you are in the habit of painting till its good enough. For a competition piece that is not good enough, you need to practice to a competition level to get better if you want to level up from bronze to silver or silver to gold. I remember reading about a Canadian painter (and he has been at ReaperCon in the past, but his name escapes me) who went to his first GW Games Day in Canada and decided the next year he would win the slayer sword. He succeeded in doing that, he also spent almost 400+ hours researching, painting, striping and re-painting that piece. He may have started off that year as a newbie painter but by the end of it he was master. Most of us don't have that kind of time, but it does show what can be done if you are up to the challenge and are willing to put in the work. There is no magic bullet to getting better except to keep painting and improving.

Edited March 15, 2018 by Heisler

[Arc 724]

Thank you for posting this.

[Doug's Workshop]

It's worthwhile to note that judging any form of art has this problem.  I come from the world of dance, and I've been privy to similar conversations.  The Olympics has their problems judging things like figure skating and gymnastic routines.  Judging anything in a subjective manner has these issues.  It's not like we have mini-painting death match (and no, that's not a challenge for next year's ReaperCon!). 

 

Heisler has written a really good piece, and hopefully competitors will take the words to heart. 

[Cranky Dog]

I didn't participate in the competitions this year. But I do have a question:

 

- Does the detail level of the mini itself have a noticeable impact? In this sense I mean comparing a Reaper metal cast mini vs a Reaper Bones mini whose details are not as pronounced or sharp as its metal counterpart. Some of the original Bones 1 models (i.e. the Noseless Horrors) are literally disfigured right out of the box. In other words, what if you got an ugly mini?

 

And I'm also grateful to see everyone else here on the boards discuss their own results and what they learned:

- Push the contrasts even more! No, even more than that. MORE, I SAY!

- Consider the light source when taking the contrast into account. Usually brighter at the top of the model, and darker on the ground level.

- Smooooooth blendings.

- And you've just mentioned thin paint fares better than thick paint.

Edited October 31, 2016 by Cranky Dog

[ub3r_n3rd]

Very interesting, Thanks for posting this Heisler. A lot to think on for us who enter reaper or any other competition.

[Sanael]

Well put, Kris.

 

As someone with a fair bit of experience in both judging and being judged in a variety of arts, little of this is a surprise...but it's incredibly good information to have laid out in plain terms.

 

I think the only time I've disagreed with any judging I've received at Rcon has been...well, every time I've entered, the judges have chosen a piece other than the one I expected them to judge. But the scores I've received have been pretty in line with my expectations.

 

Thanks for putting this up for the forums!

[DixonGrfx]

Hmmmm.......

 

You mention training the judges how to score an entry. I'm curious if you have a specific set of painted figures you use to visually provide an example of each "level" of painting for the judges to reference while training or during the scoring of the entries?

[Heisler]

I didn't participate in the competitions this year. But I do have a question:

 

- Does the detail level of the mini itself have a noticeable impact? In this sense I mean comparing a Reaper metal cast mini vs a Reaper Bones mini whose details are not as pronounced or sharp as its metal counterpart. Some of the original Bones 1 models (i.e. the Noseless Horrors) are literally disfigured right out of the box. In other words, what if you got an ugly mini?

 

And I'm also grateful to see everyone else here on the boards discuss their own results and what they learned:

- Push the contrasts even more! No, even more than that. MORE, I SAY!

- Consider the light source when taking the contrast into account. Usually brighter at the top of the model, and darker on the ground level.

- Smooooooth blendings.

- And you've just mentioned thin paint fares better than thick paint.

 

This is a tough question. A well cast and detailed piece is just easier to paint and its likely just going to be easy to achieve good results on. Its really hard to do competition level work on a piece that is less well defined casting. The round about answer is yes, you can do well with an ugly mini but you have to be willing to do the work and its going to take more work than a well cast piece. Beyond the dragons there were a number of Bones minis in the competition that did very well this year. Except for possibly the youth entries you probably couldn't have told the difference unless you knew about it. I have scored 1st place trophies with some less than stellar historical miniatures over the years so it can certainly be done.

[MissMelons]

Pfft, a poorly molded model is a great opportunity to be creative. Have a really ugly dent in the thigh of a giant from the metal not settling? Instant scar.

 

I wonder if there are kudos for cover ups like that.

[Heisler]

Hmmmm.......

 

You mention training the judges how to score an entry. I'm curious if you have a specific set of painted figures you use to visually provide an example of each "level" of painting for the judges to reference while training or during the scoring of the entries?

 

Its not training in the same way that you are thinking. Its more about making sure they have an open mind about techniques and the quality of work. For instance Banshee's blending style (a European painter for those of you unfamiliar with his work) is quite loose and is not the same smooth blending that so many of us try to achieve. If we had figures to consult against we would have to select an artist to paint those figures and then we would be boxing ourselves into that artist's style for everything. So the whole thing is rather nebulous. Or we would have to go to some kind of check of system like GW uses to get through their first cut judging. Did they use more than three colors on the base, did they use more than three colors on the mini, etc. If I had example figures I would have to have one for almost every technique you could think of, that would be a lot of minis!

 

I think the hardest thing to train or teach  judges, especially when we first introduce this judging style at Reaper Con, is that there can be more than one gold medal. There is that first impulse to equate gold with 1st place and that is not the case in an Open competition. If there are 49 gold medals in the room then we will give out 49 gold medals. We can reward those painters that have shown the skills to paint at that level and not have to narrow it down to the top three, when we do that then we have to look at what a painter did wrong more than what they did right. We have to enlighten judges to the concept that just because they don't like a technique like drybrushing that it is still a valid technique and when its done well it should be rewarded. When you are wavering between a bronze and silver or a silver and a gold either go with the higher score or have a consult on the piece with another judge. Formal training is just not an option for us which is why we like judges to be instructors first and see how they can handle a class of students before we throw them to the wolves.

[Heisler]

Pfft, a poorly molded model is a great opportunity to be creative. Have a really ugly dent in the thigh of a giant from the metal not settling? Instant scar.

 

I wonder if there are kudos for cover ups like that.

 

If you covered it up that well, how would we know? So no, no specific kudos on something like that unless a judge was familiar enough with the piece to know what kind of effort it took to get a medal with it. And if you entered it in painters that would really fall into the prep work side of things and wouldn't have a huge affect on the score anyway.

 

Any by the way those percentages we show are just guidelines to keep the judges on track. So if a judge know a piece was a tough to paint up its still going to be judge, primarily, on your paintwork not your prep work.

[Kuro Cleanbrush]

Thanks a ton for posting this, Kris!  As a first time full-judge this year (after serving my apprenticeship last year), I can really say that we are all trying our best not to let any of our own personal prejudices negatively affect our scores; we want to see folks leveling up!

 

As far as techniques/styles go, I may say something like "too much gloss sealant is detracting from this mini" or "the drybrushing appears a little chalky" or something else along those lines, but pretty much every technique/style (I'm lumping them together on purpose) does have its place if executed well.  The trick is, I'm always leery of the "that's just my style" statement, which can sometimes come across as an excuse to use a familiar (but maybe not most appropriate) technique rather than a valid artistic defense.  Heck, I feel like I do the same thing at times when I may choose to avoid adding some wear and tear to a mini of mine because "my style is bright and happy".  Is that really the case, or am I just scared of "messing up" my mini with a possibly very appropriate technique that I feel I don't execute as well as other techniques?  What I am trying to say (maybe not very well...) is that every technique or style, when executed well, can and should receive its due affirmation and approval (and the judges want to give that!), but we all need to keep pushing ourselves to broaden our skill sets so that we can have the right "tools" ready for the "right" applications (understanding that art is subjective and that there may be many "right" tools and applications).

 

Hopefully that whole post has slightly more clarity than a puddle of mud...  

[Reaperbryan]

And I can confirm on the tech side that the database of scores are remarkably consistent. I can find no instances where judges disagreed by more than 1 point. So 3/3/3, 3/3/2, and 3/2/2 are all fairly common variations and patterns in the scores, 1/3/4 occurs nowhere. And I find 3/3/3 and 2/2/2 to be by a fairly large margin the most common results.

 

The judges have all been pretty well trained to understand what a 1,2,3, and 4 should be, and the degrees of variation are all within reasonable variation for something as subjective as art.

[skippen]

Great post.  We can all be frustrated, but since both you and Michael are local, I have eyes into the system better (being taught by Kris how to judge for local comps here) helps.  I see things I am sure others do not simply because I have been taught what to see.  Though I still miss it on my own mini at times.  Though this last time, I was happily surprised and disappointed all at once.

[Corporea]

this was my first year as an apprentice judge and it was really fun and interesting!  I'd say after looking a lot of models, I started to get a sense of level of work, regardless of technique.  it was good for me to see what the judges look for and how the process goes- very educational. My mental numbers matched up most of the time.  I found I tend to be on the more critical side, which surprised me 'cause I'm a pushover in real life, but I didn't actually vote on many models- two maybe as the alternate. I guess it's because as a perfectionist, I always see the things that I can improve and that must carry over to judging.

 

I know the process can be stressful for the recipient.  We all want to be rewarded for our best work, and it can be frustrating when our hard work doesn't seem to match up with the award.  I've found feedback from the judges fantastic.  It helps me so much to know what they're seeing that I'm not.  heck, I cornered poor Derek at the end of the night when he must have been exhausted and asked him for feedback on my gold-winning tiefling.  because I want to get better.  I also worried that since I spend so much time on the forums, that this would interfere with my ability to be objective.  But since I see and often try to help out with most of you all, I guess it ends up cancelling out!

 

I guess what I most want to say is know that the judges are people who want everyone to both feel good and most importantly learn and improve.  Know that we're pretty much all introverts and when we give feedback and help, we're probably nervous!  And as the night goes on, also probably exhausted...