Thursday, October 22, 2020

Online Remote Training: Lessons Learned

By Llewellyn Falco and Clare Macrae.

Here is what we learned about how to do effective online training, to help you run better courses yourselves, or to help you choose better courses to attend.

TL;DR: Take advantage of being able to spread classes out over time. When meeting virtually, don’t pretend that you can replicate what you do at in-person training. You will need to scale back and make accommodations for the added problems of remote training.

See it for yourself at our November 2020 class “Testing Legacy C++ Code effectively with Approval Tests


Over the last 6 months, we have been intensively learning how to create online training with a similar level of quality to in-person training. In this post, we share what we’ve learned so far. 

Note: Skills vs knowledge. There are multiple types of training: many are focussed on transferring knowledge. But we believe that one of the big advantages of training is the ability to transfer skills. Teaching skills requires a lot of hands-on work from the participants. This is a different, and slower, process, but the one we are talking about today.

Benefits over in-person training

Fortunately, there are some advantages to online training, which can make them better than in-person courses. Let’s start with the positive.

Multiple Shorter Time Slots

If you remember back in school, we didn’t do a week of arithmetic, followed by a week of literature, followed by a week of physical education, followed by a week of art. This is by design. Humans learn better by doing a little bit every day. Part of this is how much we can learn in a given day, but the other part has to do with the need for periods of rest in between. Time in-between is where we can absorb and incorporate those skills into our being. Remember, exercise does not build muscles, it destroys muscles. It’s the rest afterwards where the muscles rebuild.

Spreading out a course of short periods on multiple days is never done during in-person training courses, because these typically involve the time and costs of hotels and planes. So instead we batch-up into a full day or a full week, with 8 to 40 hours of training. Sometimes we try to spin that as a good thing, with a title like “Boot Camp”.

We have found that, rather than teaching an 8-hour class in a single day, it is better to teach an 8-hour class, split up into 2 hours per week over 4 weeks. This yields much better results for the students. It not only gives them more time to absorb the information, and revisit it if needed. More importantly, they get to try out these skills during their normal activities.

Other bonuses are that it also makes it easier to maintain focus for the entire session, and even makes it easier to deal with time-zones. (I can easily wake up early for 2 hours every Monday without it disrupting my entire week. If I have to wake up early every day, my entire week is shot.)

This is true for both attendees and instructors.


There are lots of benefits to homework, in deepening an understanding. Unfortunately, it is not practical to assign homework after an 8 hour day, that has to be completed by the next day.

With short, weekly sessions, homework is now practical, useful and accessible.

We have found these useful types of homework:

  1. Repeat the exercises that we did in class, to make sure that you can understand them, and get them working on your own computer.
  2. Homework that expands your absorption, such as actively trying to use the techniques you are learning, in your day-to-day work.
  3. Supplemental reading and videos. Use this sparingly.

You cannot make homework essential, so we do not try to introduce new concepts via homework. This means that if a concept is introduced in supplemental reading or a video, we will always re-introduce the concept in class, as though it is the first time students are seeing it. This is important to not disadvantage students who don’t have the extra time to spare.

A powerful motivation for doing this week’s homework is the fact that other students did last week’s homework. Be aware that even with very motivated people, some will still not do the homework because other important things took priority. So you do not want to shame anyone for not doing the homework, but at the same time, you want to make visible that the other students are. We have found this to be a better motivator than your instructor telling you to do your homework or grading the homework.

To this end, these are the things that make it easier for the homework to be done:

  1. End each session with a clear and consistent assignment. It’s easier to do something if it’s clear what you have to do.
  2. Begin each session with a check-in, where the students can see that (usually) the majority of the other students are doing the homework.


Of course it’s not all better. We have found some serious constraints when running online courses.

The need for video

One of the things we noticed is, as an instructor you are often helping people to accomplish tasks. Without video it’s almost impossible to tell the difference between “I’m not doing something because I have no idea where to start” and “I’m right at the edge, ready to jump, and i just need a little bit of encouragement to move”. These are huge extremes, and how you help an individual student at these moments greatly depends on where they are.

We have found it to be so important that we have made video a requirement for attending our courses.

Protocol: At the beginning of each course we state that, if for any reason you need to step away from your computer, that is OK. Simply turn off your camera and mute your mic, and we will assume that you are out of the room.

Smaller classes

While it is possible to run a lab with 20, 30 or sometimes 40 people in a room, we have found it very difficult to handle class sizes greater than 12 online. The issue here is the inability to split the instructor’s attention. If you put people into break-out rooms, you are effectively blind to everyone, or blind to everyone else but those in the room you are in. If you keep them all in the same room, you can’t do small-group work. This is not the case in the real world, where you can see the whole room, and the tables can just focus on themselves. Some of this can be mitigated by having an extra person co-facilitate, but this also complicates things. You need to have smaller classes.

Have backup options

“Anything that can go wrong will go wrong” (Murphy). Things go wrong all the time, even at in person events. But I’ve never been in a physical class where I just disappeared. Online, I have.

You can lose the internet, your computer can die, you can be region-blocked, the student’s computer and internet can die, servers and services can go down. What do you do when this happens?

The best advice we have is to have backup options. Two instructors means one of you can lose the internet (if you make the other one a co-host). Having multiple channels of communication (Google docs, email, Zoom, remote-desktop to shared computer) not only means that you have a way to communicate when something goes wrong, but also makes it easier to figure out what went wrong, when the inevitable happens. Also have a second internet connection (usually a phone). 


Backup options don’t work if they are not set up and tested. We have found that we need to make a checklist of things to do before we start each class. There are usually only 4 or 5 items, and we usually won’t remember to do all of them without the checklist.


Allow time on both sides

In the real world, many instructors and students arrive early to the room, and sometimes they hang around after the class has finished. But because of the ability to instantly teleport into a Zoom call, we have a tendency to start an 11am call at 11am, and to book a 12 o’clock call right after. From the instructor’s point of view, we recommend that you open the call 10-15 minutes early, to allow people to arrive at leisure, and to ensure you can start on time. Also, clear your schedule for another 30-45 minutes after the class, to you to allow questions and discussion with students. 

We also recommend that our students don’t book anything for 30-60 minutes after the class. This isn’t so that they can have discussions with us (although that’s a benefit). It’s because they’re tired after the class, and they need a break before jumping into whatever their next thing is.


An advantage of the real world is that while you are in class, the rest of your life is out-of-sight and out-of-mind. This is not true in the virtual world. The cognitive load we all feel from the rest of our household weighs on us, and we need small releases at regular intervals. For example, it is easier to tell a child “wait 10 minutes and I can take care of it” than “wait 2 hours”. We recommend a 5-10 minute break every hour, without fail.

When we are running breaks, we usually display a count-down timer in a web browser, so everyone knows when to return. Just Google “5 minute timer”.

Need for Activity

You are simply not as charismatic and engaging on video as you are in person. Remote calls need more activities, and shorter periods of you talking. This is good advice for skill-based training anyway, but is 10 times important when training remotely. We suggest breaking things down into small exercises, that you can start almost immediately. We try to use frequent changes in how the students are interacting, to introduce variety and hold engagement.

Not Seeking Volunteers

Asking for volunteers tends to unintentionally exclude a fair amount of your class, and you may find one or two voices tend to dominate. This is true in-person, but it’s significantly worse in virtual classes, where it is easier to hide, and harder to jump in.

We suggest using systematic methods of selection. For example:

  • Rotate in order: from a list
  • Randomly select: the key here is make the random selector visible. For example, assign everybody a number, and use an online random-number generator.
  • Mechanistic selection: break into groups of two, and the person with the shortest name will go first, and then you will rotate.
  • Call on people who have not yet contributed.

Requirements and Geographic Locations

It’s easy to think that now we are virtual, anyone from anywhere can attend a course. But this is not always true. Many software products and web sites are blocked or not available to certain geographical areas. Our best suggestion is to have a very clear list of requirements to attend the course, and let the attendees determine it for themselves.

For example, our current course lists this at the top:

Course Requirements

To attend this course, you will need to be able to use the following:

  1. Zoom, with a microphone and camera.
  2. AnyDesk
  3. Google Docs
  4. Basic understanding of C++ (creating strings, creating call functions, variable, loops, etc…)

Cloud-hosting and AWS

Having people work on virtual environments in the cloud is a huge help, for remote training. This is not as doable for in-person training, as reliable internet is not something that you can count on at a conference, but is something that you have to rely on, for people to attend a course virtually. To learn more about this, check out “How to setup a Windows AWS EC2 instance for remote mobbing and pairing

See for yourself

If you would like to experience how we are doing virtual training yourself, you can always sign up for our November 2020 classTesting Legacy C++ Code effectively with Approval Tests

Alternatively, if your company would like a private course, we offer them in Java, C#, C++, JavaScript and Python, on a variety of topics relating to Agile Software Development and Code Quality, such as Unit Testing, Refactoring and Legacy Code.

Contact us at: and 

Wednesday, July 29, 2020

Modified STUK storage to fit under a SÖDERHAMN Ikea couch

With the pandemic, I've been spending a lot more time in my house. One of the things I've been watching to entertain myself is the tiny house videos on youtube. While I don't think I would want to live in such a small space, it has gotten me thinking about the space I do live in, and how to make it better suited to me and less cluttered. And, of course, it's gotten me thinking about storage.

In my home office I have an ikea SÖDERHAMN couch.
And I couldn't help notice that there is a decent amount of space under it that I could use for storage. Especially of the things I don't use very often. But with a height of just 5.5 inches there isn't many options I could find.

At Ikea, the salesperson did say that the STUK storage would work. I should have doubled checked, but I was tired and just bought them.
Unfortunately, it's too tall. But having already gotten it, I decided to look into modifying it to fit. My first hint came from the assembly itself.
I realised I could cut these to shorten it. It turns out to be easy to do with i sharp knife. Cut the top down the grove. Then bend it backwards and slice the knife between to fully separate it.

Now the only thing left to do is the 2 panels that are sewn into the walls of the box. For this I just cut them out, but I found it work better to cut a bit away form the edge so there is a little flap to hold it in when you replace it. 

Then just reassemble and it slides nicely under the couch.


I think the edges could be nicer, also more permanent.  They sometimes slide out depend on how I'm holding it, but since it's mainly out of sight, this isn't something I'm worried about or planning on fixing.  Likewise, the cuts to the fabric could have been cleaner, or repaired. but since no one ever see this, I'm not bothered.

Wednesday, March 25, 2020

Recovering a lost code base

Twice in my career I have realized that the code that was in production at a client did not match the code that was in source control. Of course if you have any type of automated deploy process this will not be the case, but if you have a manual deploy it can be.

Yesterday, Simon Cropp told me of an amazing technique to recover from this. I am writing this in the hope that it will never be of any use to anyone, ever :-)

First, let's go thru my history of what I did.

The First Time: we found the code in the closet.

The first time this happen was over a decade ago. We were fortunate that the company was being sued and a court order forbid the destruction of data. The employee in question had been let go a month or two before but their computer was in lockdown storage. We were able to find the code and check it in to source code control. This is the ideal solution. We get everything we needed. The original source.
We were lucky. Very lucky. The code hadn't been modified, since they weren't there to modify it. There was only 1 place the code could be because that module was only worked on by them. We didn't have to guess between versions. And, we didn't delete the code.

The Second Time: we decompiled the code

The next time we weren't as lucky. This was a VB.NET project and the latest in source control didn't match what was in production. Was it very different? Who knows. The employee in question had also been let go and when we realized this we ended up just decompiling the code that was in production. This creates very bad Visual Basic code, so we took advantage of it to create reasonably bad C# and move languages. This was nice, but we lost a lot of intent that was in the original source code. It would have been better if we could have gotten it back. However, we cut our losses and moved on.

The Next Time: decompiled to detect and recover the source

Here is a better way. It's a mix between the first two.
  1. Decompile the production source
  2. Also, compile and decompile the possible sources you have found.
  3. Compare the decompiled sources with source control.
    1. Maybe there's a match? 
      1. Yay! Then you have got the original source or it's equivalent 
    2. Maybe you have partial matches?
      1. Create a new code base of patches from the matching places.
      2. Repeat
    3. Maybe you have a section with no matches
      1. You can try to reproduce the code manually
      2. or, you can copy and paste those sections into the new code base and small sections of decompiled code in large sections of original code.
      3. repeat

Tuesday, November 19, 2019

Thoughts on CodeRetreat

Last weekend I‌ attended a Code retreat as part of global day of code retreat. I‌ love code retreat. I’ve been attending since 2010. I’ve organized and even facilitated some. But I‌ was rather disappointed in this one. I‌ wanted to write down the issues I‌ saw, not as a criticism of code retreat, but in the hopes of bringing back the original feel of the event and to hopefully grow more of them. As such, I‌’m using each issue to better understand what I dislike and propose a change that will improve the experience.

What it is

CodeRetreat is a full day of practice. It focuses on TDD‌ and design and you do the same problem over and over again in 40 minute bursts. Often there are small variations to help explore new areas of learning. There is often a small retrospective in between each session.



To be clear, I am a HUGE fan of pair programming. However, there is a skill to pair programming and many at a code retreat have never practiced it before. Having the 1st session default to the style where the driver (person at the keyboard) is typing and thinking tends to lead to a session where the navigator (the person not at the keyboard) is watching. It makes it hard to rotate, and discourages tring new languages.
Solution: I would have the 1st session be strong style pairing. The difference is that the person at the keyboard “is not thinking”. This makes it easy to work in a new language (you just type for the other person). It maximizes communication and helps people to connect. After the 1st session. I‌ would introduce the traditional style of pairing and open it up as to which they prefer.


Retros are a good way to learn from experience, but there are two issues with the current style. The first is we don’t really have a shared experience, so listening to each other talk full of misunderstandings. This is amplified by the deletion of code (more about that later). So that it was hard to get value out of the retros and they felt more like stand up status meetings.
Solution: I‌ would advise a two phased retro. 1st the teams spend 5-10 minutes retroing their experience by themselves, then they choose the best insight and do a group share of a single insight. I would also include in this group share the language and number of tests written.
Example group share: “We worked in java & wrote 9 tests. We were surprised by how hard it was to work around passing back booleans until we changed to passing in the if/else blocks to the method.”

Deleting Code

Code retreat has a rule that you delete your code at the end of a session. This is a rule that gets violated all the time, but has a good intention. It’s designed to make it safe to experiment. I believe in that safety. But I‌ also believe in the ability to review and learn from the code you wrote. Deleting it prevents reviewing and sharing.
Solution: First, wait until after the retro to delete the code. Second, make the deleting optional to the group, but allow anyone in the group to have the option to delete the code. In other words, unless it is unanimous to not delete the code, the code gets deleted.


I’m surprised there isn’t at least one session of mobbing and a option for mobbing on most sessions. Although, I‌ wouldn’t recommend it for the first session as it cheats people of a chance to struggle.
Solution, I would offer mobbing as a option.


Test driven development is still very new for many people. I’m surprised by the number of pairs that don’t write any test even though it is stated as a constraint. I think you need sessions just to build this skill. Probably many sessions. In the retros it was common to hear they hadn’t written test.
Solution: Slow down on the addition of constraints. Talk about the testing cycle. Ask about the tests. Give people paper so they can write the test down before code. I would even make a place to post the paper to share.


Constraints can be a great way to explore new areas of design. But too many remove all usefulness. This is used a lot in Code Retreat so I’m going to address them individually. But by the 4 session we were doing game of life with: no talking, no primitives across a border, and no if statements. We had not mastered any of these and most people didn’t actually follow the restrictions. It also massively discorages using a new language as it’s an advanced design session.
Suggestion: make constraints optional, give a variety of choices. Encourage not using them if you are doing this session in a new language. The new language is the constraint.
Constraint - Primitives: This is open to a lot of misinterpretation. It’s designed to address coding smell of primitive obsession. Which means you shouldn’t pass around numbers and strings, but objects that have a meaning. Like Money instead of a double, or Name instead of a string. However, there is an interesting issue of constructors: is the code “new Money(100)”‌ allowed? and there is a easy misinterpretation that the specific language definition of primitive is in play. This could just be an autoboxing exercises in java. Use Boolean instead of boolean, Integer instead of int.
Suggestion: have examples and a slide of the constraint.
Constraint - Muted Ping pong Pair Programming. This constraint removes communication in pair programming. It’s rather masochistic. I don’t understand why you would do this other than to show some of the pain, which can help learning, but contributes to the general issue of making the day less fun. Since these are usually saturdays, this should be the opposite of what you are striving for.
“The most important part of practice is making sure you want to show up for practice tomorrow”
Solution: remove this constraint entirely. At a minimum add it a possible minimum constraint

New languages

One of the things I‌ loved most about code retreat was trying new languages. This time that was hard to do because of the amount of other constraints that where in play.


I also noticed that we ended with less people than we started with. This is amplified by the fact that my partner and I‌ were the only 2 people that didn’t work for the company that hosted the code retreat and I live in a very programmer heavy area. Given the number of programmers in the bay area, there should be multiple code retreats and all of them should be overflowing.
I have had sooooo much fun at code retreats. Practice should be fun. I‌ feel that some of that has gotten lost in translation.
Solution: Bring back the focus on fun. Make it a stated objective.

Tuesday, April 2, 2019

The Problem with Hackathons

Today I went to TestBowl 2019 - Software Testing Competition.
I had some reservations going into it, but it was in my neighborhood and only 3 hours so I thought "why not".

It was disappointing, but the thing is, I've been to 40-50 practice events for programming. I've hosted quite a few of them myself. So why was this different?

This blog is an exploration into that question...

The Event

The event had about 20 to 30 people. I believe all of them were at the conference anyways (meaning I was the only local). The had dinner and did some questions, that were stated to be ice breakers but didn't include things like actually talking to other people. There was some interesting trivia questions though.
After that we broke into teams of 2 and where given a Test Target (a government website) and a Bug Tracking system (from the company that sponsored the event). This was a contest, so the points went to the most bugs submitted (this is an simplification, but basically true).

We used a mindmap to log as we went and only entered the results into the tracking system in the last 15 minutes.

We tested for around 90 minutes. Then it was over and I went home.

What was wrong


One very possible reason I didn't enjoy this very much is that it was just exploratory testing. While I admire exploratory testing, it doesn't bring me the joy that programming does. While I do not believe this to be the reason, I think it's important to acknowledge this bias in me as it could color my impressions.

Conversely, I do think it is this bias that is allowing me to see the issues in this event more clearly. It was lacking the "spoonful of sugar" making the distasteful parts more salient.

Contest considered Harmful

The single most harmful thing about this event to me was that it was a contest. The reason I enjoy, attend and participate in hands-on practice sessions is the learning. That's always where the value and focus is. The contest changes that focus and obscures the focus.

For example, there was the option to work alone. I think some of the other teams or even most of the other teams worked separately. This is much worse for learning, but it is understandable that people would prioritise that when "the goal" is number of bugs in 2 hours.

Let's imagine working alone. It's rather hard to see what you would learn if you don't have anyone to learn from. But even with just 2 people there is less chances to learn tricks you can use later. Also, there weren't constraints which are usually helpful when learning. Forcing you to work in a new way and discover new techniques.

Alternative Motivations  

It's obvious that some of the motivation for the hosts was to get people to try out their software. This isn't inherently bad, learning new tools is often useful. However, because contest was drawing attention away from learning we used the tool but without really learning it. I didn't come away with ways the tool could help me to test. This is something that could be addressed better if the focus was solely on learning.

No Retrospective

So another downside of the contest is there isn't much sharing between teams. There was sharing of the score, but this didn't pass along learning just increased focus on the scoring and task. The competitive aspect tends to prevent sharing and reflection . The goal being "to win" instead of 'to learn" also means that when it the work is over you are tempted to leave because it feels like the event is over. You don't stick around to learn because that wasn't the point.

No Space for Stretching

"Hard in training, easy in combat"
Another element that was missing was the space to try new things. Humans tend to either be in a state of practicing or preforming. Part of fitting into the time limit was prioritizing for getting everything done in the time limit. This left little space for actual practice.

In conclusion...

The reason I enjoyed the other practice sessions is the hands on learning. I hadn't realized the amount of effort that was put into making sure this was happening. 

Monday, December 3, 2018

Safeguarding: A step-by-step guide

By: Llewellyn Falco, Josh Widzer, Jay Bazuzi

Bugs happen. While you could simply fix them, you could instead take an extra step to prevent similar mistakes from occurring again. This 25-minute process will do that.

We will go in to the philosophies and reasons to do this in other articles.
1. When to do it
Do safeguarding right after you’ve fixed a bug. The same day or next is good. This is when it’s fresh in your mind and when improving the system still feels relevant.

The key roles that need to be in the room are the people who:
  1. understand what happened and why
  2. wrote the bug
  3. detected the bug
  4. fixed the bug
  5. project manager (someone who can approve the time required to work on the fixes)
  6. might resist the proposed remediation

2. Root Cause Analysis (RCA)
We are going to gather impartial observations about what happened.

Create a Google Doc that everyone can access, with with the following tree:
  • What caused us to write the bug?
  • Why didn’t it get caught sooner?
  • What made it hard to fix?

Everybody is going to start adding nodes under these three headings. They will also add questions in response to the nodes. All of this is done at the same time by all attendees without any talking.

This section is timeboxed to 10 minutes.

3. Vote
We will do a version of dot voting. Everyone will vote on as many items as they want but no more than once per item. Voting is done by putting your initials at the front of any item.

  • What caused us to write the bug?
    • [JW, LF]  requirements were unclear
    • [LF] Name of a function lies about what it does.
  • Why didn’t it get caught right away?
    • No automated Tests
    • [JW] Hard to write automated tests for this section of the code
  • What caused debugging time / cost?
    • Logs were too verbose, so we didn’t see what was going wrong.
    • [JHB, JW] Hard to redeploy site

After voting, copy the top 3-4 items into a section labeled: Remedations
This section is timeboxed to 3 minutes.

4. Budget
Before we come up with solutions, start by asking: was the total impact of this bug small, medium, or large? Have everyone hold up 1, 2, or 3 fingers. Pick the most common answer. Then propose an initial time box:
  • Small = 1/2 person-day
  • Medium = 2 person-days
  • Large = person-sprint.
Since we execute solutions immediately, this is the point where we need the approval of the project manager.
Because we intend to do 3 solutions, each solution can only be ¼ of the total budget (to allow some slack). This means that each item will be budgeted to:
  • Small = 1 hour
  • Medium = 1/2 day
  • Large = 2.5 days

This section is timeboxed to 2 minutes.
5. Identify Remediations
Next we are going to brainstorm improvements to our system to reduce the chances of this issue happening again. Going back to the Google Doc, everyone will silently add ideas to the second section that we copied from the top brainstorming section.
Solutions that require extra discipline are bad solutions. We are looking for ways to make success easier.
Remember to keep in mind these are timeboxed solutions meant to improve our system and environment as opposed to solve everything. Often small improvements yield big returns and if the problem still persists, we will get another chance to do a safeguarding in the future.


  • Remediations
    • Requirements were unclear
      • Bring users into our grooming sessions
      • Earlier usability tests
      • Do earlier demos
    • Hard to redeploy site
      • Write down checklist of deployment steps
      • Automate build/test/upload sequence
      • Get a second server to deploy blue/green
After 7 minutes, everyone votes again just as we did in the last section.
This section is timeboxed to 10 minutes [7 brainstorming, 3 voting].
6. Add items to task board and do them!
Because safeguarding already has time approved work on them immediately.
Nothing we’ve done so far matters if we don’t implement any of the solutions. Make sure that useful action comes from this exercise, immediately add them to the taskboard (budget has already been approved in step 4) and start working on them. Remember that these items are timeboxed and are not meant to completely prevent the problems in the future, but rather to lessen the chances.
Final Notes
It is important to remember that safeguarding is a skill. The first time you do it, be patient and give yourself extra time, maybe an hour. Also, remember to practice it regularly, usually once per week, as you will get better at doing the process and finding good remediations.

Special Thanks to Arlo Belshee for creating Safeguarding.

More at: