Doing Quality Assistance Remotely

“We’re here to make great games for our players – with high quality virtual worlds that link millions of people around the world. Anywhere, anytime.”

Our company mission is the driving force behind our operations, and building high quality into our games is clearly a key component of it. Our processes, workflows, and tools have been under constant development over the years to continuously achieve this. With the current situation in the world, InnoGames has switched to having employees work remotely to help contain the spread of COVID-19. It would then come as no surprise that we have seen even more changes to our Quality Assistance Engineering approach.

Change is always scary, but at the same time essential for transforming an existing matter into something new and better.

The Road Here

We do Quality Assistance in many ways similar to how Atlassian does QA, with placing the QA Engineer in the middle of the development process and involving them at every step, as opposed to the traditional way of having assurance at the very end of the process. This approach has proven quite beneficial for us and over time we have made tweaks to further adjust it for our own environment and needs. However, parallel to our optimization of our QA workflows we have also been building-in quite a few communication dependencies for our QA Engineers.

When you actively include someone in your full development process they need to take part in quite a few meetings, discussions, brainstorming sessions, demos; they get to be there for all the stops and milestones along the way. In an open office this is easily achieved by having the QA take a central position, offering them easy access to everyone and all conversations. Switch your whole operations to home office though, and you cut away this possibility. With all products continuing their development as normal, this asked for quick adjustments.

Logistics

Before you can set sail to new horizons, you have to make sure your ship is in order. For us this meant ensuring we have the tools to work remotely.

Device Pool Coverage

A particularly good thing for testing purposes was our ability to distribute our test devices, to provide a decent coverage spread across different people. While there are emulators that can imitate various devices and provide a way to inspect matters, it doesn’t quite replace real device coverage.

What we have realized is that now more than ever we can support each other between departments. That specific OS setup? That super low-end device? This unique device that seems to have issues with your app? If you or your team do not have it, chances are someone on another team does. Between our personal devices and the distributed test devices we can form a similar type of coverage that we had before. A simple question and a small favor between departments can quickly resolve a potentially brain-racking challenge.

Company Servers

If you require being connected to your own network, VPN connections on devices come quite handy. Android has a wide range of free VPN client apps like strongSwan which let you set up a network profile. iOS offers this directly in their Settings, under Profile → General one can locate the option for creating a VPN configuration.

An alternative to this would be setting up your VPN configuration on a computer, and then simply sharing the internet connection via Wi-Fi to the device of your desire. No matter the OS you’re operating with, there’s a possibility to do a setup. MacOS has this option under “Sharing” in System Preferences. Windows has it under “Network Connections” where you can right-click to check the properties of any network and allow sharing under the “Sharing” tab.

You can also work out something similar using proxy server applications such as Charles. It might be best to try out multiple options to figure out what is the most stable solution for you.

Daily Work

As QA we tackle a wide range of tasks on a daily basis. For many of these the environment has no influence and they can take place remotely just as on-site. For others the same cannot be said and require some tinkering to make them work.

Real-Time Demos

A big part of our QA process are the Demos, essentially a showcasing of the implementation of a story accompanied with a discussion between the present parties and a decision if the story is complete. In the office it’s easy to gather everyone and quickly get some feedback and a discussion going. For remote work we use multiple approaches.

Screen sharing within a video call allows for everyone to observe the changes and simultaneously have a discussion. Thankfully a number of communication channels, such as Slack, already provide this as part of their options. The tricky part may be getting to share the screen of your connected mobile devices – but there are ways to solve that too:

• For iOS devices once you’ve connected and trusted your device you can use Quicktime to record the screen and display it on your Mac.
• For Android devices you might wanna try out a solution like scrcpy where with a simple “adb connect” command you can not only reach your device but get a full display and control of it on your machine.

With some communication channels the screen sharing option may already be there, as an example see Zoom’s instructions.

Turned-Based Demos

Everyone being at home means our work dynamic has very likely shifted. We have to spend time with families, take care of children, prepare food, and do an array of other tasks that could usually be skipped when you are in the office. The result is that our availability may not be quite the same as it is on a day in the office.

To combat this it is nice to explore alternatives to real-time demos, whenever possible. For example visual changes do not necessarily require an immediate check. Providing a screenshot of changed UI, or a video of newly implemented animations, could suffice. Not only does it give people time to respond but you might even find it useful. It gives an opportunity for everyone to think more on the topic and maybe provide even better feedback than usual. It is quite like playing turn-based instead of real-time strategy games, as people are able to plan their next move!

Testing Dojos

Despite checking user stories individually, sometimes issues occur at an integration level. It can also be the case that only when multiple elements are combined together and you interact with the content as a whole that you can provide feedback on it. That is why we have testing dojos, where the whole team can play and test together.

In the office the QA would try to set up everything ourselves, allowing the team to focus on the most important – the testing. Because the same assistance is not possible remotely, we have set up quite some guides and instructions to enable everyone, including those outside the development department, to set things up themselves.

What we put more focus on is moderating the actual dojo. Doing playtest video or audio calls with a group of people requires some form of a process to let discussions and questions flow smoothly. This can be as simple as ensuring best practices are being carried on: having video on (to give awareness when people want to talk), muting yourself when you are not talking (to avoid disruptions and noise), making sure the conversation stays on topic.

An alternative can be splitting people into smaller groups and forming multiple different calls. Just as you might separate into groups in the office, the same can work online!

Workflows

Similar to daily work, you may find that not all of your product processes translate easily to remote work. Nothing scary because when there is a will you can build yourself a way.

Bug Handling

One obvious aspect when it comes to quality is the handling of bugs for your product. Remote work demands new ways to ensure transparency of the product quality status and continuous tackling of incoming bugs. We do this in multiple ways, depending on the team preferences.

• For our team stand ups we are using new JIRA Dashboards to ensure we are on track. This involves filters with queries such as “AND (resolved >= -1d OR resolved <= endOfWeek(-1w) AND resolved >= -3d)” to fetch JIRA tickets from the previous working day.
• We are utilizing our very own Slack bot Boggy to get information on incoming bugs, as well as daily updates on the bug status.
• We have set up long-term metrics overviews, providing the usually available statistics from our weekly QA reports also for a longer time-frame, to better show trends over time.
• We are also revisiting our overall Bug Strategy for some of our products. It might not sound like the best time for bigger-picture viewing and modifications but it can actually be the perfect time. A change of environment gives you a new perspective on your usual flows and could easily point out areas for improvements.

We have also kept our usual reporting of the product quality, such as via weekly reports and bi-weekly product reviews that involve the whole team and other interested parties from the company.

Left: Boggy Slack posts of incoming bugs and daily summary. Right: daily stand-ups JIRA dashboard.

Work Times and Schedules

An interesting discovery on some of our teams is that time works quite differently when you work remotely. The people that showed up first in the office are suddenly struggling with mornings. The people that enjoyed late working hours are suddenly first ones there in the morning. Some people need breaks during the working day that they might not have needed in the office. Regardless if the reason is family, lack of commute, or simply the change of environment, the result is the need to change your work set up.

Something we have done at Forge of Empires for example, is moving our daily Beta deployments to a later time in the day. “Beta” is a server we open to all our players, where we introduce new content some time before its official rollout. Anything merged during a day ends up on Beta the next working day. Even postponing deployments just an hour or two might give people the time they need to feel more comfortable with their daily business.

We have people who outside of any meetings have switched to working mostly in the evening, or have split their work time in morning and evening. As a company we already practiced trust-based working hours, though for the most part we kept our core hours the same. To some degree this still holds true, yet we have seen people utilizing flexible working hours more and more, and with no impact on the product operations. You just have to keep your communication strong!

Communication

What is usually a strength of Quality Assistance suddenly appears to become its weakness. Needing to maintain the same level of communication and assistance as you would in an on-site environment sounds like a daunting task. But it turns out all you need is some bravery to make a change, and the ability to ask for feedback.

Filter Your Intake

Very early on we noticed what you will find suggested elsewhere on the internet – make sure your communication is effective. And for good reason, following various threads and channels all day long is exhausting. Think about it in terms of a physical office and you would quickly realize you would rarely attempt following multiple conversations at the same time, or being in multiple meetings at the same time. It’s overwhelming and quite ineffective.

The simplest favor you can do your brain is tuning out the channels which are not needed for you. Being informed is surely nice, but so is the ability to focus on tasks and discussions, to be able to work in a creative and/or analytical fashion. If something is important enough for you to know about, it will surely make its way to you.

Writing vs Calling

Rather than writing on and on, it is often better if you just do a quick call with all involved parties to sort out the details. This way you can ensure that everyone is actively involved in the topic and not distracted with something else. On top of all that having video improves the communication, it lets you experience its nonverbal aspects (gestures, facial expressions, eye contact, etc.).

Of course writing is still the most effective when it comes to maintaining the longevity of the information. You can always write a summary after a call has concluded and provide it to other channels or save it somewhere where it can be referenced in the future.

The key is knowing when a simple question or topic has come to a point where a call becomes the better approach. The length of a thread could be a good indicator, but an easy option could be to just ask the other participants if they feel a call might be a good idea.

Meetings and Events

In general we have upheld our meetings in the development process. Maybe you see a pet or two more, or a piece of clothing less on your colleagues, but for the most part meetings work remotely just as on-site. Even for retrospectives or planning, there are many online whiteboard possibilities letting you keep the processes almost the same as before.

We have however seen that some meetings work better for different environments. For example while there have always been interesting presentations and discussions in our QA Forums, there is now more experience sharing as every team is somehow in the same position. Suddenly the different technologies used or the products being in different stages are less noticeable.

At the same time meet-ups like breakfasts or coffee hangouts do not quite feel the same when done remotely. It is however possible to obtain the same value via different means. Such gatherings we have replaced with virtual alternatives, like gaming evenings.

Summary

While our Quality Assistance approach heavily relies on direct connection with many people and departments, we have seen it is still possible to maintain it in a remote environment. With putting focus on open communication and knowledge sharing we were able to optimize our processes to continue efficient daily work. Through this period we have also picked up lessons that we can continue applying for a long time in the future.

There is a simple lesson here – nothing has to stay the same. And that should not be scary at all.