James Montalvo

Editing while executing

2023-11-28

Tags: EVA, Maestro

“Fix EVA procedure authoring” was the founding purpose of Maestro. Handling execution of procedures was not, until it became clear how intertwined it was with editing.

Ensuring a consistent editing experience

When someone runs an EVA training event alone (without a full flight control team, as is typical in planning phases), adjustments to the plan are managed by scratching out and penciling in steps on the paper procedure. The trainer has full flexibility to change any aspect they want to achieve training and planning goals. The purpose of these training runs is to make adjustments and optimize the plan.

When a larger team is involved (e.g. for a real EVA in space or a larger simulation), we still need to make changes to the procedure during execution of essentially every EVA. This is very different from many other procedures in space flight. You wouldn’t, for example, adjust the procedure for a rocket launch midway through. Even for non-life-critical procedures inside the International Space Station, we don’t typically change procedure steps midway through. Why EVA procedures are different will be the subject of a later post.

When we make changes during an event with a large team (again, a real EVA or big simulation) these changes can’t live as pencil marks on one person’s paper; they need to be communicated, which can be complicated depending on the magnitude of the change. There are really three types of changes made: First, sometimes just an audible change on the MCC voice system is acceptable (e.g. “Skip page 5, step 10; crew already performed it earlier”). Second, if there is too much content for an audible, we may need to write out the changes in a [Flight Note] like:

  • Delete step 10
  • Move step 3 after step 5
  • Add step 13a: Retrieve wrench
  • Change step 17 to stow hardware on handrail 1234 instead of 4321

We would also include rationale for these changes in such a Flight Note.

The third and final option for a real-time change is to modify the existing procedure and resend it to the team. This is time-consuming (editing in Word) and can lead to confusion (“which page 2 do we use?”), and therefore is rarely done.

While considering what modern procedure execution might look like, it became evident that all three of these could be replaced by just editing the procedure in Maestro. Rather than making an audible call like “Move step 3 after step 10”, instead someone could just drag-and-drop the step and allow everyone to see the change in their own time rather than expect them to hear the verbal call. The Flight Note method is essentially just a written-down version of many audible changes, so editing in Maestro could replace this as well. And of course the third option was modifying the original procedure, which editing Maestro is.

As stated at the top of this article, executing procedures was not a founding purpose of Maestro, but it became clear early on that editing would be an integral part of execution. If a procedure author is working on a procedure in their office months before an EVA, the pressure is off, and their expectation of what the authoring tool needs to do is potentially lower. Of course everyone wants an easy and fast experience with every tool they use, but, as stated in [Why Maestro?], when “seconds matter and human life is on the line, [Maestro] must be exceptionally easy-to-use”. Editing and executing could not be designed and built separately. If we had built Maestro purely as an editor first, and then later tacked on execution, we would likely either have had to substantially change the editor to support execution later, thus disrupting the user experience, or we would have been stuck with a suboptimal editor.

Building both

We’re a small team, and we had no option to ask for additional funding or developers. By choosing to build execution capabilities at the same time as editing we were also choosing to delay both of them. To minimize this delay, we decided to make the minimum viable execution capability. While we wanted to make editing and execution collaborative (i.e. multiple people making changes), we opted instead for single-leader functionality: One person could broadcast what they were doing. This greatly simplified the problem. Instead of dealing with conflicts between multiple people changing things, we simply published the leader’s application state to the server and shared it out to anyone who wanted to follow along. Choosing this method made it possible to have procedure execution capability very quickly, leading to many benefits like learning how the application had to be restructured to facilitate execution, understanding how editing would be used during execution, and forcing us to make editing fast and easy. It also gave us feedback from users on what was missing from procedure execution. Of course we got feedback like “I want more than one person to be able to edit and check-off steps”, which we had deliberately not built for the MVP, but we also learned many things which I’ll explain in the remaining paragraphs.

Since printed procedures would still be produced as a backup and as a method to use the preexisting review/approval process, and Maestro produced a Word/PDF document output identical to the web-based view, there needed to be a way to talk about the procedure in a medium-agnostic way. Historically steps were referred to as “Page 5, EV1 column, step 7”, but on the web there are no page numbers. Additionally, activities in the timeline were simply named like “Egress and Setup” or “Pump Module Install”. Flight Controllers did not want to have to verbalize the 10 syllables to say “the Pump Module Install activity”; they wanted to simply say “Block 5”. So to solve both problems we prefixed all activities with a number in both Word and Web. Furthermore, we stopped referring to them as “activities” and called them “blocks” to save 3 syllables.

We also got feedback that we needed two editing modes. When a trainer is running a test event early in the planning phase, with just a few people interested in following along, then the editing-during-execution experience can be virtually identical to the editing experience when not executing. The editor wants full power to change anything in any way. When leading execution of a larger event, however, there need to be some limitations. For example, when adding a new step under the current step 7, it cannot be numbered step 8. This would cause the original step 8 to become step 9, which would then make the Word/PDF/printed versions of the procedure out of sync with the web procedure. We considered calling this new step “Step 7a”, as that is how it was often done with Flight Notes and other methods of changing procedures in the past, but in Maestro substeps are given letters, so substeps and execute-added steps would conflict. Instead we decided to name such steps like “Step 7+1”. This (a) wouldn’t conflict with anything, (b) would be very obvious that it was an added step, and (c) would be minimal additional syllables when spoken on the audio loops. These steps were also stylized differently, so they showed up as something added real-time versus something that was scrutinized during the pre-EVA review process.

The pressure of an EVA creates a situation with zero-tolerance for failures. Conversely, during normal low-pressure usage of Maestro, users might find a bug and shrug it off, never reporting it. As such, post-EVA we get many bug reports that could have been reported earlier. Using Maestro during EVAs both puts the pressure on the program as well on the users, and helps improve the program faster. Furthermore, being under a microscope has made the Maestro team much more demanding of quality: we write more and better tests, we manually test exhaustively prior to releases that will be used in EVAs, and have built automated load testing to simulate EVA-level loading anytime.

Finally, we got feedback on the benefit of Maestro for procedure execution. We knew in our hearts that it would be useful, but had no proof until it was actually used on a real EVA. As the first EVA using Maestro came to an end, I was proud of how well it had performed and knew we would be getting a lot of positive feedback, but was completely shocked and overwhelmed with how positive the response was. But that is a topic for a future post.