Delivering Creative Feedback: Critique Skills

Acknowledgement

Presented at ACM SIGGRAPH 2012 Courses and the ACM SIGGRAPH 2011 Business Symposium

After spending about seven years in games, I took some time off and decided it was time to regain some core drawing skills that had been withering since I graduated with my BFA in the mid 1980’s. I had the great fortune of enrolling in a one-week intensive drawing workshop with Tony Hull at City Lit in London. As this was a traditional drawing class, numerous critiques were held each day, and I realized that despite many years of sitting in critiques, I had rarely seen anyone conduct such a positive, useful and truly constructive critique as Tony had.

Tony had the unique ability to focus conversation around the work in the most positive and constructive tones I had ever experienced. Unlike most critiques I had been part of, Tony was interested equally between the work, its intent, the critique and the environment of the critique itself. I then realized that despite working as a professional designer and then a CG artist, I had not sat in a single, “proper” critique session since transitioning into the entertainment business. A few pints later, a great friendship was born, and I was able to subsequently able to bring Tony in for some drawing and critique workshops with the crew of a feature I was working on.

While Tony is still the master teacher of these workshops for artists and designers in production environments, we have continued to collaborate on the format and content since the first one he did for my crew in 2004. For the inaugural SIGGRAPH Business Symposium (SIGGRAPH 2011), I designed a companion version of this workshop specifically for business leaders and entertainment executives. This course for SIGGRAPH 2012 and its 2011 predecessor are largely due to what I have learned from Tony and for that I am deeply grateful, for his insights, wisdom and generosity. For more information on Tony, his teaching and his fine art work, please contact him via his website: www.tonyhull.mac. (Update 2025: Keep an eye on Tony’s page as he is working on a book on Critique and its use in work and life.)

Introduction

So much of the SIGGRAPH conference is firmly rooted in unequivocal scientific or mathematical fact, whereas the other half of the work presented is in the subjective realm. While the data, or “facts,” are open to thoughtful debate inside and outside of the conference, it can be much easier for people to offer feedback on those less subjective topics. However, when the content is subjective and based on emotional interpretation, “right” or “wrong” observations are more elusive (as they should be).

“…when subjective feedback is vague and hard to act upon, collateral damage likely affects trust and morale.”

The idea for this course started several years ago during a product review, when a VP of production in a very successful game studio told his executive producer and game designer, “I was playing the build [game] last night with my 7-year old son, Marc. He thinks your gameplay sucks, and I agree.” One morning, a few years later in dailies (on a feature), an Academy Award-winning VFX producer exclaimed to a room, “That shot is crap. The lighting is wrong, the angle is bad, the color is tragic…” before walking out of the room, leaving behind a shell-shocked crew.

At a studio level, when subjective feedback is vague and hard to act upon, collateral damage likely affects trust and morale. Done well, a critique can be the oxygen every production team needs to grow and help artists advance their craft and core professional skills, all while ultimately benefitting their clients, directors, users, etc. This course is intended to provide a simple framework that can be adapted to any production environment. While the terminology used in these notes is artist centric, everything contained here can be easily applied to business meetings and engineering/code reviews.

Workshop vs. Lecture

The very first iteration of this course was more of a lecture format. After reviewing the initial results, we quickly realized there was too much emotional distance between the “students’ and the course’s message. When students don’t have skin, or in this case ego in the game, the lessons lacked impact regardless of the participant’s role in the organization. After we switched to a hands-on workshop where students drew about 40-50% of the time and practiced critique for an almost equal duration, the connection between lessons and the techniques being taught became readily apparent to the students. (Due to the logistical constraints of SIGGRAPH, this course will be limited to 10-15% hands-on.)

To keep the course focused on critique technique as opposed to drawing technique, it is essential to level the playing field in the course itself. If your class is filled with people who can draw, have them draw with their non-dominant hand (left hand for right-handed artists and vice versa). For those with artistic ability and/or artistic professionals, you can have them draw (preferred) or alternatively have them use digital cameras and laptops to create the work samples. (Note: if you go this route, make sure there is no network access.) Typically, our materials are large Post-it style easel pads and inexpensive charcoal with paper towels, with all participants completing 5-minute black-and-white, continuous line-drawing exercises.

Exercise 1

Pop Quiz: Which is the better photograph and why?

How do you answer such a question that is vague and lacks context? When offering your opinion in a professional situation, your ability to offer actionable and effective criticism is dramatically increased when you have, at a minimum, a brief or goal that you can assess the image against, as well as an understanding of the state (condition) of the work being evaluated.

Exercise 2:

Setup: If possible, compose a simple still life in the center of the table that your students are sitting around. Anything will do. Typically we have a speakerphone, a bunch of coffee cups and everyone’s cell phones stacked in a pile. Bonus round: Add a desk lamp to create shadows (however, it’s entirely unnecessary).

Exercise Part 1:

Each person takes 5 minutes to draw the still life you just assembled (with their weaker hand, see above) as a continuous line drawing (they don’t lift the pencil from the page). All students should do the same exercise together.

Exercise Part 2:

After everyone has created his or her drawing, the next step is discussion. In groups of 10-12 people, put each drawing up on the wall (or if sitting around a table, turn the easels inward so everyone can see all the drawings). Start with the biggest/tallest person’s drawing, and have the group provide feedback as to what they like and dislike about the drawing. Repeat this again with the person on his/her left, and continue until everyone has received feedback.

Evaluation

Normally the feedback portion of this exercise starts with some awkwardness but then starts to get smoother. However, the feedback generally tends to be scattered, inconsistent and is not actionable.

As was the case in Exercise 1, the feedback was solicited in vague terms, which more than likely resulted in unfocused feedback. Ambiguous, subjective, inconsistent and/or thoughtless feedback delivers no practical benefit.

“Don’t Take it Personally”

During the evaluation portion of the exercise, students quickly realize that receiving feedback is not something anyone looks forward to. Of course, in the work place, when a colleague or direct report gets critical feedback on a piece of work they did, people are quick to remind them to “act professional and not take things personally.”

In doing so, we dismiss the fundamental reason the vast majority of people became artists in the first place. For most people, it was precisely so they could personally find a way to contribute to a creative body. No competent director would ever dare tell an actress not to take his or her feedback personally as it’s widely understood that acting is a deeply personal act of bearing one’s soul, or imagination, to an audience of strangers. While they may be contributing to someone else’s vision, the work they contribute is still a deeply personal interpretation of what they understood the need to be.

(Note: If you are reading these course notes and did not participate in a workshop, this concept will seem rather obvious, but without “skin in the game,” it’s hard to appreciate the significance of this point.)

Context is Key

If the goal of giving criticism is to provide understandable, useful and actionable feedback, the context by which this criticism is delivered is everything. In the second exercise above, the assignment was vague, as was the feedback you gave each other and, ultimately, took away. Because the assignment lacked a Creative Brief, the viewer is unable to determine if the drawing was successful given the vague context of the brief. Likewise, it becomes difficult for the reviewer to offer feedback that can help improve the drawing towards that goal. Without a clear and focused brief, the feedback will be arbitrary and less likely to be viewed as constructive by the artist.

In thinking about delivering feedback, a brief need only establish context for the intent of the work. The person asking for the feedback should then describe the state of the work and the feedback being requested. Once this is understood, the reviewer can deliver feedback that is relevant, useful and appreciated. Each time someone (re)establishes context before providing feedback, they remind themselves to provide thoughtful, constructive and focused remarks.

Too often, people provide feedback that is either purely subjective or that comes with little real thought. Thoughtless, unactionable feedback is distracting and an enemy of a great creative environment.

Context: Roles

In the previous exercise, and in the absence of a clear boss, producer, etc. in the group, we started with the tallest person in the group to lend the air of intimidation to the exercise. This subtle dynamic reminds participants how people communicate differently when they are aware of the role, seniority, rank, etc. of the people they are addressing. The rank or weight of people’s position is always apparent and unfortunately, will be a significant yet unintended factor in any feedback situation. It is human nature for employees to want to please their bosses. In creative environments where peoples’ livelihoods are tied to subjective opinions and criteria, it is natural for people to seek reassurance whenever it is available (i.e. the GM/MD is walking across the production floor and an artist shows him what he/she is working on.)

The left is a manager’s view of their team dynamic; the right is the employees’.

As is the case in relationships, people often want to be heard but don’t want their problems solved for them. The greatest perk of seniority is the privilege of silence, or specifically in these situations, to not offer a solution. It is often far more helpful for people to walk through the problem at hand by asking questions and letting them develop the potential answer and “own” the solution.

The greatest perk of seniority is the privilege of silence, or specifically in these situations, to not offer a solution.

Context: Direction vs. Opinion

To state the obvious, in large creative productions, there are many different heads with different responsibilities. Obviously, a director or game designer will own the vision, and their supervisors are responsible for working with the crew to get work delivered that reflects their vision. Conversely, the head of a VFX or game studio will have responsibilities to his shareholders and/or partners that may appear to be at odds with the direction a director has delivered. To the average production artist, this is rarely relevant.

i.e.; A junior artist is working on a shot that came without many reference materials. One evening he shows the shot to a studio head and asks what the studio head thinks. The studio head replied he really “doesn’t care for the red on the house in the background”. Wanting to curry favor with the studio head, the junior artist later changes the red to white, only to be told to change it back to red in dailies the following day.

In this example, the studio head inadvertently undermined the supervisor on his payroll. Had the studio head asked what the direction was for the shot, he would be reminding the artist of what the requirements are and could frame any feedback in that context. If he still wanted to share his opinion and frame it as such, he could do so and still redirect the artist back to question his own work against the direction he was given. i.e.: “I’m not sure what direction you were given for this shot, but I don’t think the red is paying off. Most importantly, that’s my opinion and you should confirm the actual direction with your supervisor.”

The Creative Brief

The central theme of this course is to eliminate as much subjectivity from delivering feedback as possible and keep feedback focused on the work being done. In product design, industrial designers typically approach their work as solving a problem. This approach of framing the assignment as a well-defined problem to be creatively solved is the key to making feedback less personal and less subjective.

Most projects in computer graphics and software development have (or can easily establish) a set of clear and precise technical constraints and specifications. (In many cases there will also be a market requirements document.) That specification document should be partnered with an equal document that focuses purely on the subjective or emotional requirements of the project. The specification and the brief collectively serve as a concise and clear definition of the work to be done or the problem to be solved. This latter document, the creative brief, will serve as a means of defining the creative problem to be solved for the duration of the project and, more precisely, as the highest level of context for evaluating the work.

Writing a good, effective brief consistently takes longer and more iterations than expected. A good creative brief is not a feature list or specification. (Typically, the smaller the number of authors, the more likely a clear, strong brief will emerge that serves for the duration of the project.) While less is more in authors and a strong creative brief, it is critical to establish consensus and understanding amongst your team. (A template at the end of this piece is one proven format and methodology, but adaptation is encouraged.)

A Framework for Delivering the Message(s)

There are always examples to be had of the creative genius who crafted their opus as a solitary effort, but they probably weren’t in the design or production of a modern film, game or TV project. Large projects require large teams that are invested in collaborating towards a single vision. As mentioned here repeatedly, the tone and approach used to deliver criticism directly impacts how the feedback will be utilized. With that in mind, here is a framework for effectively delivering feedback, regardless of the environment:

1. Be on brief. Request and listen to the brief and the state, then contain your feedback as to how the work does or does not meet the requirements.

2. Be focused on the work and the task at hand. Limit your comments to the work in front of you and direct your comments toward the work, never toward the artist, their intentions, their personality, etc.

3. Be present. To state the obvious: Don’t split your attention between the critique and reading e-mail, sending texts, etc.

4. Be constructive. Above all, remain objective and remember that critique is not about a reviewer’s likes and dislikes, or the approach the artist should have used. Objective critique is about discovering opportunities for the work to meet or surpass the requirements of the brief.

5. Be clear. Describe what you see and the impact it has. There is no need to provide solutions or to speculate.

6. Be honest. Your colleague, employee, etc. deserves to hear honest opinion, and it is their responsibility to filter the feedback and decide what to act upon. Providing hollow praise or withholding honest criticism only damages the level of trust on a team.

“Plussing” Technique

Walt Disney created the term “plussing” for a method he developed at Imagineering to encourage designers to always take each idea “one step beyond”. This philosophy closely follows the premise many design school faculty hold that no idea is ever perfect and can therefore always be further improved. Conversely, in improvisation theater, the words “…no, but…” are never to be used during improv work, as they shut down conversation. Performers are instead taught to treat each line as a gift and respond with “Yes, and…” to advance the work.

Applying these two principles in concert creates an approach for critique that encourages discussion and expands ideas with the notion that listening to dialogue and adding thoughtful feedback helps advance the work. The language used in improv, specifically “yes, and” highlights the value of carefully considering the language used to describe and evaluate work. Start critiques by building on what is succeeding against the brief. Some good words for plussing include:

• likely

• effective

• successful

• achieving

• working

• paying off

• positive

• strong

Some examples:

• “The scale and form of the car are working quite well against the brief”

• “The character’s silhouette is very effective”

• “I think the smaller props are really successful”

In any relationship, starting a conversation with “You look terrible” or “That shirt is awful” is neither likely to start a good conversation nor encourage good will. This does not imply one should avoid delivering critical feedback or walk on eggshells around unsuccessful work. Starting with what works will always be the easiest part of any critique. Delivering critical feedback is likely less pleasant, but good critiques require clear, actionable feedback. (Remember when you were receiving the feedback on your sketches.)

By its very nature, critique is a very personal discussion. The center of discussion is work developed with significant emotional investment by the artist and the feedback being delivered is very subjective. Establishing a conversation where “work” is evaluated against the “brief” eliminates a potentially toxic dynamic of “I” or “me” evaluating “you.” Having established this, work can be evaluated with great precision against the brief, increasing the effectiveness of the comments offered. Examples:

• “I think red is the wrong choice for this level” as opposed to “The brief calls for this level to be very dramatic. The red does not quite succeed at that.”

• Instead of a vague statement, “I hate that gun design… it’s terrible,” try “The silhouette of that gun is inconsistent with the character’s brief and overall design.”

While it is human nature to want to help solve problems, the reviewer’s obligation in critique is limited to offering observation and feedback. There is no need or expectation of offering solutions to a challenge, and it is best if the reviewer does not offer any. It is essential that the artist own the work they are doing and they remain invested in any potential solution they choose to implement.

In many situations, it will be more effective for a reviewer to ask questions that lead an artist to a particular point. For example, instead of telling an artist, “The car should not be blue, I think red would work better,” it is likely to be more helpful if the reviewer asked the artist, “Have you considered a much warmer color for the car?” or “What other colors did you explore for the car, and why did you eliminate them?” This approach makes it clear that there is a question around a particular aspect without providing the solution.

Often it is the case where the reviewer knows something is not succeeding but will have difficulty in precisely expressing what is not working well. Precise language is the goal, but it is not a requirement. When this happens and a questioning approach does not work, borrow a technique from drawing instructors: They tell their students to draw what they see, not what they think. Likewise, if the reviewer describes what he sees, not what he thinks, it will still be useful. It is rarely useful for a reviewer to speculate on intent or ability. If a reviewer is not certain or clear, then it is best to simply explain that. Remember, the role of the reviewer is to offer critique, not solutions. Above all, remain considerate in how you deliver feedback, and deliver a direct, honest critique without being harsh.

Keeping Things Cool and Easy

Critique is a dialogue, often a deeply passion-filled discourse in which the stakes are higher for some people than others. Heated, passionate and even contentious debate can be healthy provided the debate is always about the work and never about the artist or their intentions. When in doubt, keep discussion centered on what is presented and how it relates to the brief. Explaining or talking through ideas can be useful, yet having people defending their work is likely a sign of trouble. Blaming (or covering for anyone) is toxic and unproductive. Avoid lengthy discussions about the past (“how we got here”) and remain single-minded as to what can be improved now and going forward. When in doubt, find ways to bring the discussion back the brief.

Finally, senior people need to be self-aware that their status comes with implications and (often false) assumptions that can be counter-productive to the room. Senior staff must remain acutely aware of the implicit weight their comments carry, regardless of intention. Keep feedback related to the brief and be:

• Careful in approach

• Constructive in tone

• Considered in observation and language

Failing Fast in a Critique Culture

Building environments that foster innovation and embrace a “fail fast” mentality is certainly in vogue right now. Too often, executives confuse creativity with something that just happens “organically” in environments where things are “free to happen.” Failing without learning is simply wasting time.

In reality, most successful creative organizations have developed specific processes around their crafts that serve as the underlying foundations for great creative work to happen upon. Every organization talks about supporting innovation, open dialogue, etc., yet few actually build an atmosphere that continually feeds oxygen to small creative sparks. Developing creative briefs, communication skills and feedback loops are the foundation for building an atmosphere that encourages innovation. Converting this into a core habit is essential.

When supervisors in a studio collectively decide to embrace and encourage critique on a regular basis, a culture begins to form. Obvious issues will be the need to find a regular time and location for it, and establish what the structure for that studio should be (i.e.; how many pieces are reviewed each time, who owns uploading/assembling the materials, how long each person has, protocols, etc.). Less-obvious issue will be gaining visible support from supervisors and production management. It is essential that all supervisors attend critique (to set the example and demonstrate the import of critique), production management does not schedule meetings against critique and, most importantly, people are not pulled from critique to finish “just this one thing I need.” When deploying a bottom-up approach, groups will need an ambassador to explain the process and, most importantly, the benefits to the executive team (the goal: convert executives into participants).

Cultural change is certainly easier to implement with executive sponsorship, and nothing beats an executive team that demands critique become part of the normal planning and production processes. Aside from the obvious benefit of knowing they are scheduled and supported at a studio level, when executives participate, transparency of communication is established in the studio and the trust starts to grow. When trust increases in the studio, people will find it easier to ask for and provide feedback. Trust improves communication and morale across an organization.

A Suggestion for Structure

Each studio and team must find its own methods and structure for conducting critiques. After about two years of experimenting, my art supervisors at Microsoft’s now-defunct Fasa Studio eventually developed this very effective structure.

• Duration: 1 hour max.

• Content: 3 x 15-minute-long reviews of specific aspects of the game (particular level, hero characters, vehicles, lighting, etc.), presented by the lead for each aspect.

• Format: The lead starts with the:

  • Brief: 1-2 minute overview of the brief for the work

  • State: 1-2 minute update as to the condition of the work being assessed and what work is NOT being assessed. (i.e. Layout is locked, textures are done, animation is temp, and lighting is preliminary. Translation: No comments on animation or lighting, please.)

• Review: 10 minute max, presentation of materials and group discussion on the work being reviewed.

• The Wrap: The lead will review the feedback received, followed by these two critical steps:

  • Passes: The presenter will repeat the feedback he is NOT going to use and explain why. Arguably the most important step in the critique as it publicly affirms that the particular feedback is not being ignored, yet will not be acted upon, and the reason(s) for that decision.

  • Takes: The presenter sums up the comments they will act on and describe, as necessary, how they plan to approach the revisions.

• Agreement: The art director or design director agrees or disagrees with the wrap. If in agreement, the next presenter will start their session. If there is disagreement, the presenter and the art director will discuss it until they have an agreement, and then the presenter will do the wrap one final time to ensure everyone is on the same page.

Standups ≠ Critiques | Dailies ≠ Critiques | Playtests ≠ Critiques

Many studios mistakenly believe they hold critiques in the form of dailies, rounds, etc. or smoke tests, and play tests. Unquestionably these regular exercises and reviews are essential to the business but serve a particular need of the project, as opposed to the needs of the people creating the work. Critique is a tool for staff to improve their approach, their perception and their own skills. Clearly, that ultimately benefits an organization and its clients; our goal is to grow a collective body that advances the quality for all the work that it creates and delivers, regardless of client or project.

Concluding

As budgets get tighter and the costs of failure increase, every studio needs to find ways to ensure it delivers the highest quality work in the most efficient manner possible. SIGGRAPH is one of many incredible opportunities to leverage research and technology to advance computer graphics. Critique is an age-old, fundamental skill that offers studios a low-cost, low-risk methodology to leverage its people to raise their craft and the quality of the work they collectively deliver.

Asset management provides the ability to closely track the lifecycle of each digital asset. The cost of tracking each asset and even the ability to monitor the percentage an asset is “technically complete” exists. Despite this, the ability to know if employees and contractors are emotionally engaged remains elusive. A transparent feedback cycle ensures junior and mid-level people have a safe, structured environment in which they can offer their opinions while providing a unique opportunity to learn by example how to better communicate and absorb creative feedback. Likewise, senior staff can continue their own development by constructively challenging one another while encouraging debate and divergence of opinions.

Project Name Creative Brief

v 0.9 , Date

The X Statement:

A single sentence that describes the super-objective for your product’s long-term vision. (Remember, this is not about features but about the emotional connection you want to build between you and your user.)

Executive Summary

Two paragraphs (max) that describe the overall creative goals for the project and the emotions you want users and reviewers to experience and ultimately take away from playing the game. This is not about technical accomplishments and/or specific features, nor should you describe your game as a combination or alternative to other games.  It’s about the core mechanic, core emotional arc and fiction that will be the common basis for the product’s gameplay and visual and brand identity.

“Bocce Ball Assassin” is a new and exciting genre that combines the old-world charm of traditional Italian lawn bowling with the intense feeling of power and sheer brutality by using blunt-force trauma to  eliminate the men who date your sisters--in keeping with the oxymoronic idea of killing people while lawn bowling. The game will focus on a few characters who are deceptively charming and comfortable, and very believable.  The game will rely on a visual style that is very graphic and typical of a comic book that always feeling more illustrative than photoreal.  Etc., etc., etc….

Primary Design Goals

• Three bullet points: These primary goals are keyword attributes that the design must achieve

• Less is more:  As you will refer to these points regularly when providing feedback or as a means of framing critique, the more brief you are, the easier it will be to become consistent over time.

• Emotive:  This is not about technical limitations or features, but rather the emotive responses you want to elicit in your users.

(Optional) What It’s Not:

• One to three bullet points: If it helps clarify your project, include a many as three single bullet points that clarify what your product should not be.

• Avoid cliché: Don’t employ widely used or easily misinterpreted terms, like exciting, dramatic, warm, inviting, clean, etc.  That doesn’t mean you can’t use them, but try to find terms that really describe how you want a user to feel about the app.

Some Notes to Keep in Mind:

• Great products make a very simple, clear and concise promise to users and then over-deliver on that promise. Think of this Creative Brief as the core promise.

• This is not a specification or a technical document; those are likely needed and should be a partner to this document.

• Avoid the “F” word: features or a feature set. This document is to describe the emotional connection you want your consumers to have with your product.