How to Get Real Value from Every Review Gate

Hello everybody and welcome to our office hour series. It’s good to see some folks back and welcome to everybody who’s here for the first time. So today we’re going to talk about the review date. So not whether to run one, everybody here runs them. Today’s about what happens to a review after that meeting ends. So what actions are we getting from this and how can we improve? So, you know, mostly everybody has sat through a red team that kind of cost the whole team a day and changed sort of nothing in the draft as a result. So the comments got written, you know, meetings happened and two weeks later, you know, gold team finds the same problems and has less time to fix them. So the one idea I want you to hold on to as we work through things today is the review is only worth what gets resolved afterwards. So what are we actually actioning? So we’re section through a red team the whole way from an invite that says almost nothing to a closed log where every finding has an owner and an outcome. And you will leave with the slides, of course, and included in that there’s a kind of a little bit of a planning template that you can take away as well. So, quick housekeeping before we jump in, and especially for those folks that are new. So, this is part of our Office Hours series. We meet every other Thursday, thirty minutes live. We have Q and A at the end and I also try to follow-up with folks if you have a question that I can’t get to and you’ll get the recording and the slides afterwards. The next up in our series in two weeks is shift left. So we’re really focusing on building compliance and clarity into the first draft. So review has less to catch and that kind of pairs well with what we’re talking about today. And then after that on the fifteenth, we’re talking about what does good look like. So I’ll be joined by Kyle Peterson, some of you may know him, but he will join us and talk about how customers and how folks that he’s meeting with, he’s doing a lot of on sites, a lot of work out in the field, and how are successful bid and proposal teams managing to evolve in today’s more today’s outlook, today’s pressures from an AI, from a tech perspective. So what does that look like? I just had a message that folks are having trouble hearing. Is anybody else having that trouble before I go on, or is this a specific thing? Just want to make sure it’s not on my end. Okay. Only seeing a comment from one person, so that may be on your end. Yeah, I think Amy, might be on your end. So, okay, good. Okay. So, just I think we’re good on this end. And like I said, you’ll get if you have issues hearing or have issues with the pieces, you’ll get the recording and the slides afterwards. All right. So, I’ll go ahead and jump in. Chapter Defining Review Gate Decisions So, when I think about when we talk about this session, this is kind of the argument the session hangs on. So, a review gate is a decision. So, gate should end with a call about the draft. Are we gonna approve it? Are we gonna return it? Are we going to approve it with name changes? If nobody in the room can say which of those just happened and what decision we made, we haven’t progressed anything forward. So here’s how that kind of plays out a lot of times. So reviewers get a draft folder and a date, they comment on whatever they happen to notice because they don’t have clear direction on exactly what they need to look for. Writers then fix the comments they agree with and kind of skip the rest because there’s not a lot of tracking on which is which. And then the next gate tends to find the same problems, but now we’re closer to submission. So, a lot of the resolution doesn’t happen. And none of these steps involve a bad reviewer, a lazy writer, issue on that side of the house. This is just what happens when the structure around the review is missing. So today that’s what we’re going to talk about is really dive into that structure. Chapter Review Structure and Framework So what we’re going to cover five parts, gate purpose and scope. So what each review decides and what it leaves alone. So what are we focusing on specifically in this space? A reviewer briefing. So what do reviewers need before they open the draft? Comment resolution, how feedback becomes attract finding and the disposition and status it carries until it closes. And then between gate monitoring. So the signals that a section is stuck or maybe drifting and where the platform carries the structure. So two kind of caveats, your gates may be pink, red and gold or something else entirely. So I will use that terminology, but the names matter less than the decision that each one makes. So just swap in your own vocabulary, your own structure there. And then the timings kind of as well. So, you know, forty eight hours, twenty four hours, a few days are kind of defaults for a normal schedule. Obviously, if this is a five day task order, you’re going to compress those mechanics. So we want to keep the decision, keep the criteria and disposition every finding. So those three do not compress even if the timeline does. All right. So this is kind of the loop and the pattern that most of you will recognize. So the gate is undefined. Red Team Thursday says when, not what the review is for and what it should leave alone. That’s an important piece of the process as well. So unfortunately, then reviewers arrive on briefs, so they read as themselves. So the engineer reads for accuracy, contracts reads for risk, the executive reads for tone. Nobody reads as the evaluator because nobody handed them the criteria. And then the comments land with no owner. So we have 40 margin notes, but no real decision or direction about which ones really matter. And then the writer opens the file Friday morning and just starts at the top and begins working through. Chapter Identifying Common Review Gaps And then gold team reopens the questions red team asked with that shorter timeline. So what can we do and how are we going to address these things? So why does that happen? So underneath that loop are four gaps and they map to kind of the four parts of this session. So a purpose gap, nobody wrote down what the gate decides or what each or what lane I guess you would say each reviewer owns. So, that can we can kind of close that gap with our half page charter, a briefing gap. So, reviewers know the draft exists, not necessarily the criterion it’s scored against. So, that can close with the brief being said with the draft. Resolution gap. There we’re saying, you know, comments live in the margin, so whether they get addressed depends on the writer’s memory, timelines, all those types of things. So, we can close that up with a log that gives every accepted finding disposition, excuse me, and status. And then the visibility gap. So between gates, the proposal manager sees dates and promises, but not the state of each section where we really are. So we can close that with five signals checked on a cadence that the schedule sets and we’ll talk about all of these. Chapter Worked Example Case Study So let’s make it concrete. So I’m going to take one fictional section 2.3, the quality control volume, its facilities, ODIM, recompete is going into red team. This is my made up opportunity that we’re going to take through this process so we can see what that looks like. Okay, so this is the invite. So we’re looking at the invite as the reviewers receive it. So red team is Thursday, nine to twelve volume two is in the shared folder. Please read it before the call and add your comments. So seven people received this for seeing this bid for the first time. That’s the structure we’re setting up. So look at each reviewer and what they are deciding alone. So am I judging the solution, the compliance or the pros? Which criterion is 2.3 scored against if I know I mean, if there is one, do I know? Are we addressing formatting? What counts as serious and is it my place to weigh in on some of these things? So, the predictable result there is going to be, you know, seven reviewers, seven different reviews, 40 comments, 30 of them about wording, you know, a compliance gap might get, you know, a one line that nobody really prioritizes it because it’s, you know, next to 39 other items. So we don’t have any priority or any tracking after that. So nothing in the invite is necessarily wrong, it’s just missing a few things. What does the gate decide? What does it leave alone? And which lane is each reviewer owning? Chapter The Review Gate Charter So, let’s look at the decisions that each gate is going to make. So, whatever colors you use, however you structure this, each gate makes one primary decision with a bounded set of questions that are going to feed it. So, decides whether the approach is right, compliant and aligned to the capture strategy, proceed or return. So, it leaves polish in pros alone because there’s no point of polishing an approach that might change obviously. Red is going to decide whether an evaluator would score the section highly against the criteria. So proceed or name what must change and by when. So it leaves solution redesign alone. So that was pink’s call and at least style alone unless comprehension, of course. And then gold decides whether this is ready to submit as it stands, submit or it’s going to fix name defects. So it doesn’t reopen anything red closed unless there’s a reopen trigger, which we’ll get to. This is the gate of the charter. So, you know, six lines, half a page, first half of the template. So purpose, the decision that this gate makes in one sentence, kind of tip to take, if it takes a paragraph, then it’s two gates. Then we have two kind of decisions or two sections that we’re making. What’s in scope? So, the bounded set of questions and the section M criteria they score against five to seven questions is plenty. We don’t need to overload there. Out of scope. So, what should reviewers not spend time on today? This is as important as what’s in scope and which gate will pick those items up. So, we have reassurance there to let go of that concern when we know where it’s going to land and then it will get addressed. What is our entry criteria? So, what must be true of the draft before the gate runs and the named version reviewers open so nobody is reviewing a moving document, making sure that whatever our naming conventions are that we’re all working on the same page. And if the entry is not met and the date can’t move, the gate runs narrower. We are aware some of the limitations that we have or what we’re coming to the table with. And then our exit criteria. So what must the section show before it advances so a second person can confirm it. So usually no open criticals, every major issue is owned and dated. And then who is the decision owner? One person who calls approve return or approve with changes. So the reviewers are going to advise the owner makes the final decision and then the proposal manager records it. Fill this in once per bid and attach it to every invite to make sure folks are on the same page. Okay, so here is the two point three red team with a charter. So our purpose is we want to decide whether two point three would score outstanding against M. 3.2, excuse me, and list exactly what must change if not. So what do we need to do to get there? In scope, what are we in scope here? So compliance with each L. 5.3 element, evidence behind every claim, risk to the customer’s mission and clarity for a first time reader. So that’s what’s in scope in our review. What’s out of scope? So, sentence style, acronyms, graphics, again, unless they block comprehension, those are going to go to the editorial past. And restructuring the QC approach because Pink already made that call. So we don’t need to reopen that unless it’s noted. And then, you know, what is our entry? So draft 0.6 frozen Wednesday at five is the baseline compliant to the L. 5.3 checklist inside page count every requirement traced. So that’s where we’re at when we’re starting. What do we need at exit? In order to exit, we need no open criticals. We need every major owned and dated and volume two lead signed off. So, who’s our decision owner? The volume two lead is our decision owner in this situation. Reviewers advise but the lead decides and the PM records it. So, we have the same seven reviewers, the same Thursday, same meeting, but now they open to the same version with the decision in front of them and fewer out of scope comments get written because the charter already said where to not spend the morning where we’re not spending the time. Chapter Effective Reviewer Briefing Strategies Okay, so I want to talk about kind of part two. So, the charter says what the gate decides, but the brief gives reviewers what they need to decide it well. So, good reviews scope the gate and the best ones kind of also scope the reviewer. So, let’s dig into the review. Okay, so same worked example, same paragraph, same reviews with and without the brief. So, unbriefed, we’re getting things like, you know, tighten this awkward phrasing. Do we say this in 2.1? Is this too long? Not sure this is our strongest example. All of those are reactions, not findings. So, these are what you’re feeling and what you’re seeing when you’re reading this. They’re not necessarily things we can action on. So, we can’t assign it, we can’t verify it, and we can’t tell which ones really matter and are going to make a difference in the end. Briefed, the same people write, you know, M. 3.2 scores demonstrated ability to detect and correct deficiencies. This paragraph shows inspection, but never shows correction critical. So, these are things action on and we can focus in on that because of the briefing because of the information that we came in on. So, we want to make sure that we are reading, you know, read the way an evaluator reads action. We want to make sure that we’re leaving some of the minor things for editorial pass and we’re working through some of these examples. So, there’s on the right here on the brief section, there are several examples. I won’t read the slides out to you, you’ll get them after. But you can see the difference between the two about what sort of outcomes and what sort of actions we can get when we are phrasing that differently. So, we have the same reviewers, the same amount of time, but we can sort, assign, and close those items. And the ones on the left can only be sort of discussed because they are sort of reactions and feelings. We want to make sure these are actionable and they are improving quality and moving us forward. Okay, so let’s talk about what goes in the brief. So, six parts, one page, one piece. This doesn’t have to be kind of a massive item, but we want to make sure we have, you know, our criterion quoted. So, Section M language for every section under review with the scale or weighting if we know what that is, if that’s been published or shared with us. So reviewers should read the words the evaluator will read. So what is our message to prove? So the discriminator this section carries and the proof point that it rests on so reviewers test the argument rather than rediscover it. Your lane. And so this is something that we really want to hone in on. So what is your lane and what position are you taking when you’re doing this review? So, to one person, technical credibility to another, customer alignment to capture for argument’s sake, terms of risks to contracts. We want to make sure that everybody has everybody’s owning one lane and each lane has an explicit owner. So, we don’t have, you know, seven people all reading for the same thing while nobody’s reading for compliance or nobody’s paying attention to like specific types of risk. So, ultimately, again, and if you’re on a small team, one person can hold two lanes, one person can hold three, and that’s fine. We just want to make sure that each one has that explicit owner. And then the questions to answer. So, like I said before, five to seven questions always kind of including one on conflict. So, does anything in here contradict with another section, volume, assumptions, schedule? Is there anything that we’re doing that’s contradicting ourselves throughout this piece? And sections kind of fail reviews on their own, but proposals fail because some of these sections disagree with one another. So we’re looking for cohesiveness there as well. So we want to make sure we’re looking at what’s settled plus the named version under the review and how to write a comment, which is next. We’ll talk about that. But we want to make sure that we’re setting this ahead forty eight hours at least on a normal schedule, obviously, hopefully the night before on a compressed one, that timeline can change, but we want to give time and reviewers who read the criterion beforehand arrive with a question or arrive more engaged in the material because we’re not spending that time upfront setting scenes. So we want to make sure that folks have the time to digest it as well, if possible. I understand the timeline sometimes are a little bit messy, but if possible. Okay. So let’s talk about the comment. So, the anatomy of a comment we can close and a comment we can evaluate against. So, a comment that can be closed has four parts. So, where. Where in this instance is 2.3 paragraph two judged against M. 3.2. What? So, the paragraph describes inspection frequency and checklists. It never shows a deficiency being corrected. We need a fact about the draft, not a feeling about it. So we need to say, you know, why this matters. Well, M. 3.2 scores, we want to make sure we’re scoring, detecting, correcting. So as written, an evaluator can’t find evidence for the correct half, and that is a material scoring risk in this example. So, it’s stated as a risk and not a verdict. So, unless the solicitation defines the scoring consequence, we don’t get to declare that. And then what do we do? So we want to add the corrective action example from the current contract, forty eight hour closure, ninety eight percent first time fit. So two sentences, no new page. So that’s what we need to do, how we need to address it. And then we can look at severity defined. So nobody kind of debates it in the room. So critical is compliance gap or a direct score risks. So that closes with verification before the gate exists. Major weakens the argument or evidence. It gets an owner and a date and may still be open at exit. Minor is clarity or polish. So, editorial pass, defer or reject. We’re not going to spend time on that in the room. So owner and due date get added a triage and not by the reviewer. So, and then one more ask score the section to one color per criterion, 20 comments and a green score means polish five comments and a red one means the argument is missing. So, different review when the writer comes in in the morning to address what’s coming across their plate. Chapter Finding Disposition and Status Okay. So, we’ll talk about the resolution and this intends to be the part that a lot of folks skip. So, the review will say the review produced 60 comments, feedback becomes a finding at triage. So every accepted finding gets a disposition and a status. Silence is neither. So we have to make sure that everything has those pieces with it. So talking about the disposition and the status. So two separate things happen to every accepted finding and we don’t want to blur them. So the disposition is the decision. So we want to so on this specific and we’re talking about act, affirm, reject. So an act, the finding stands and the draft will change. So one owner, one due date assigned a triage. Defer, it’s valid, but not for this gate. So it needs a destination gate and a one line on why and it stays open until it’s handled there. And then reject, the team decided not to act, it closed when the decision owner records who decided and why. The status is where the work stands. So, open, in progress, closed, and how finding how we’re closing or excuse me, I’m thinking ahead the head of talking. But how a finding closes depends on its disposition. So, closes only after someone other than the writer verifies the change and addresses the finding. Touching the paragraph is not closing the finding and reject closes on the recorded reason. And defer of course stays open until it’s gate. So that separation really makes our exit rules clean. So a critical must be act with verification, a major can be act and still open and a minor can go to an editorial pass be deferred, be rejected. So the two things we want to kind of take away from that are rejected and ignored differ by one kind of recorded reason. Ignored comes back, rejected does not. And closed reopens only on a trigger agreed to on the front end in the charter. So we want to look at things like an amendment that changes the requirements, something changes in the solution or staffing, we’re a pricing assumption doesn’t land where we think it is. So without a trigger, we’ve already closed that strand. Okay, so who moves feedback to close? So, let’s talk about kind of five handoffs. The middle one is the one that probably gets skipped the most. So, the reviewer writes the feedback and scores the section. The proposal manager triages straight after the review. So, you know, in a normal schedule within a day, within the hour, we’re on a really compressed or tight schedule. And so, this is where that feedback becomes an accepted finding. So, we want to merge duplicates, confirm severity, send back anything without a finding before it enters the logs. We want to assign those to owners and due dates. And then the owner is the one who will act, defer with the destination, of course, or propose the rejection and keep that status current and keep us moving forward. So, the volume lead then will verify that the change answers the finding and close it or approves or overturns rejections as we’re going through process. The proposal manager closes the gate when exit criteria are met, not when the meeting ends, not when we move on to a next, you know, to our next stage. So, we’re going to carry any open majors and deferred items into the next brief, so we’re all on the same page of what’s been resolved and what and what hasn’t. So the triage is kind of the, I would say cheapest step on the slide, and it does get skipped most often. So, 40 raw comments assigned as written become 40 different arguments or 40 different positions. So, this allows you to consolidate duplicates. It settles the kind of severity conversation, what are the things that we need to address most and makes clear which handful of things have to be closed before the gate exits. So, what do we have must we address? Okay. So, here and I’m getting I know I’m getting short on time and so I’ll try to get through these and like I said, you’ll get this after as well. So, here is the log for 2.3 after red team. So, requirement seven, we have inspection without correction, we have critical act. The writer owns it two sentences closed, verified by the volume lead. So we’re looking at our examples of what this looks like after. So we want to make sure on all of our spaces we have where, what we’re judging against, our finding, and then, you know, who owns this, what is the disposition, what’s the status? And what does that look like in a log? So this is just an example of what that log looks like and how we can manage that moving forward. Chapter Monitoring Signals Between Gates So the log closes the gate, but the time between red and gold is where some of those sections can stall. So we need some signals to investigate. So we want to make sure that we are signaling with our monitoring what needs to be addressed. So, let’s talk about the five signals that are worth sort of investigating. Look at what kind of what we’re seeing here. None are, you know, asking how it’s going, but they’re signals to investigate not verdict. Verdicts. So, these aren’t decisions that we’re making. So, do we have open criticals? So, any act critical open past its date is the first thing that we need to address. Days since the last edit. So, much time have we has something been sitting? That could tell us that a section is stuck or that there are additional things that need to be addressed or things that we might not have flagged in our review. From a page count perspective, so do we have unexpected growth in our page count after review? So, do we have parked content landing where it belongs or it could be additional content that nobody reviews? What does that look like if we have drastic changes in our page count? We have coverage change. So, do we have an amendment or restructure that untraces a requirement? It’ll show up there before anybody reads it. And do we have deferred items due? So, do we have items that are deferred and are there specific items that are coming up on our timeline that we need to address? So making sure that through this process, we are investigating the signals, not just holding on for the next review or waiting for very specific items. And these are a few things that we might want to look at. So put those signals kind of in one view and then we have our volume to, you know, nine days in this worked example after red team. So 2.1 is on track, we can leave that alone. 2.2 has a critical, that is two days overdue. So that’s the first thing that we need to address. You know, 2.3, our section is closed with one minor deferred to gold, excuse me, so on and on. So we have two conversations today based on our triaging. We don’t have to deal with all of these things. There are things that can wait for gold. There are things that can continue. So the monitoring is going to tell you where to not spend your time as much as it’s going to surface items that are critical to review. So, where kind of the platform carries the structure? So, this is I’m just quickly touching on from the VisibleThread side of the house. We don’t have a video today. So, I just wanted to quickly touch on the platform piece of it. So, on the VisibleThread side of it, or it could be another platform that you are using as well. But how can a platform help with the transition step? So, instead of, you know, on paper in a spreadsheet, that sort of thing. So, it’s the tagging. So if we’re talking about tagging each comment carries severity. So you can tag, you can call people out, you can pull different items. And so the log bills itself from the review. So you can filter to open criticals in one click, we can make sure that we’re managing all that in a place that’s traceable and visible to everybody in the space. Inline comments, so the comments sit on the text that it refers to tied to that trace requirement. So, there’s never ambiguity around what that through line looks like and the resolution is visible in the draft. Advanced review, so you can assign reviewers by section section and lane, you put the charter and be brief in front of them in that workspace and see open items by section between gates without, you know, having to ask for that. And so, that piece of it, I just wanted to touch on really briefly because I know a lot of the questions I get when I talk about things like this are, okay, that’s a lot of information. How am I managing that? And so I just wanted to throw a little piece in there since we don’t have a product video today that these are things that are addressed from a management and a review perspective in the platform. Chapter Immediate Implementation Steps Okay, so what are just kind of do a few things like obviously I work these are information heavy slides and if it feels like a lot or a lot of things to bring in or change, we can start here. So three things that we can do at the next gate, no tool, no platform required. One, we want to put the purpose sentence and the out of scope list in the calendar invite. So just one paragraph, reviewers will read it because it’s short. You know, we’re not overwhelming in the space. So, we just want to get that point across as directly as possible. Two, we want to send the criterion with the draft and tell each reviewer which lane they own. So, how are you showing up? What is the criteria that you’re judging against? And we want to ask for one score per criteria, not only comments. And then three, give every accepted finding disposition before the gate closes. So, we want to make sure that everything we’re either acting, we’re deferring with the destination of course, or we’re rejecting with a reason. So, want to make sure if we’re rejecting reason status and everything that we are tracing is traced back to status and how we’re working through it. So, a spreadsheet with columns is enough. That’s a great place to start and then you can continue to build out from there. So, you don’t need a big process rollout or tooling in this template. I’m going to jump to the next slide because this is the template. So, this is the template that you’ll get in the with the deck and then the follow-up. So, here we’re looking at, you know, one page per gate, four blocks. So, purpose and scope, reviewer brief, resolution log, and between gate signals. Eight checks with every timing set to your schedule rather than mine in this hypothetical example. The last two are going to kind of decide whether it survives a live bid. So every rejection carries a reason and a name and the signals get checked from the log, not from the status meeting. So if you keep those two, you’re going to keep the rest. So this is the kind of three that I showed on the last slide and this is a good ways to get started. Chapter Measuring Review Success Metrics Okay, so let’s talk about my kind of finish up here with the numbers. So three numbers per gate tracked bid to bid. So we want to make sure that our findings are tied to criterion, requirement, customer priority, a discriminator, evidence gap, on and on as a share of all comments. So we want to look at, you know, sharp strategy comments, but not do things like tighten this up or like change the wording, things like that. So the briefing is going to move that first and we want to make sure that those are actionable items. So we want to disposition at gate exit. So accepted findings carry a disposition and a status when the gates close. We want to target all of them. We want to make sure that everything at least has a disposition, even if it’s being pushed, even if it’s not being resolved. We want to make sure that we have a disposition and a status so everybody knows what we’re working against. And then repeat findings. So, comment at gold that red team has already raised with no reopen trigger behind it. So this is kind of what we focus on and it should trend towards zero. We want to make sure as we’re logging all of these items, are we reopening things that we don’t? Are we doing rework that doesn’t need to happen? And are we kind of muddying the waters ourselves without accepting the things that we can move on from? So together, all of those things answer one question. So did the review change the draft or did we just have another meeting? Did we actually make progress and move forward? So if we take the kind of baseline of what you have now, and we’re looking at kind of one person, one afternoon, one volume from your last bid. So comment volume by itself tells you very little how many comments and how much interaction. The mix usually makes the case for a charter. So what does the volume look like? But what does the quality look like? And how are we actually actioning these items? Okay. So, one idea to kind of take away, a review is only worth what gets resolved. So, if we have a review and we are not actioning items, we’re not closing items, we’re not making decisions and we’re not progressing the draft, we are wasting our time obviously. So, we want to make sure that we’re scoping the gate, but we’re also scoping the reviewer. So, we’re giving every finding a disposition. And when we do that, the draft is going to get stronger at each gate instead of moving forward in our timeline just shrinking. So, again, you’ll get the slides, the recording, and that template that I had in there. And I did have some questions. I’m going to take those and I’ll follow-up with you after because I know we went a little over time and I want to make sure that I get everybody out of here. But again, you’ll get the resources after and if you’re interested or if you are a subscriber to our sessions, our next session will be in two weeks of course on Thursday and that will be shift left and we’ll talk then about quality in our draft or higher quality in our draft and that kind of goes along with the review. So, how we can really use our reviews to then create higher quality drafts and move forward or shift left in a more meaningful way. So, will like I said, everybody will get follow-up from this. I do have the questions here that were asked in the chat and I will follow-up with you directly. And I hope everybody has a wonderful rest of the week. Thank you for joining us and we hope to see you again in two weeks time. Thanks so much. Bye.