Imagine you spent hundreds of thousands of dollars researching what users want. You asked them dozens, if not hundreds, of questions. Panel after panel shuffles in and out of your conference rooms. Your team gathers the data and deduces what users want next but time is running out.
You throw together a design that your stakeholders like but you can't gather the necessary feedback to know if it solves the problem. You have pressure to deliver something ... anything ... because you need immediate results. Your stakeholder is happy so you decide it's ok to skip user testing. 1
At this point you really don't know if this feature will have the intended user or business outcomes. You are making a bet. A big one. But instead of lowering the risk you've increased it because you completely left out the very consumer that's supposed to be at the center of the experience. You've gone from spending a ton of money and time on user-centric panels and questions to end up at a business-centric solution. 2
More often than not these decisions will end up costing you more time and money. Short term gains, long term losses. You'll have go to through an entire user feedback process before you can iterate on your designs. This time you'll be under even more pressure as you'll be facing the prospect of users abandoning your app and sales losses or drops.
Unfortunately, this is a daily scenario at most businesses. What can you do if you find yourself in this situation?
Ask for more time
In all reality deadlines can move and a lot of the time they do. If you believe a delivery date is in jeopardy or you are not comfortable with the time allotted to deliver a feature, ask for more time. You'll be surprised at what can happen if you are proactive with issues. People are reasonable and if you can make a compelling case, more often than not, you'll get more time.
But let's say you asked for more time and the request was denied. What happens next?
Understand what you are delivering
First, don't panic. You have options. The very first thing you should do is spend time analyzing what your solution is and any issues it may have. Not every solution is perfect, there will be edge cases that are unaccounted for, but shoot for a solid happy path and work your way out from there. Ideally you are working in an agile environment where you can iterate quickly once something is in users hands. The following tips will help you get feedback so that you can fill up a backlog comprised of fixes and updates for that feature.
Informal testing with colleagues
I work in a building with 300-400 people -- that's a lot of potential feedback. A lot of my colleagues are not part of our workflow or familiar with what we are doing, but a large portion of them use our apps, which is good for us. We reach out to team leads to help us coordinate some small user testing sessions when we need to get immediate feedback. To do this type of testing effectively you have to know what the most effective means of performing the test are. It really depends on what you are trying to test. We usually print out low fidelity mockups or share Overflow (see next point) journeys that we make from Sketch artboards.
Overflow User Journeys
We love Overflow. It's been our best tool for mapping out a user journey and finding any gaps in the experience. This is the cheapest and quickest way to test as you just need to create an Overflow project and import your Sketch artboards.
Once you have all of the screens in flows in front of you it's really easy to identify and fix problems as you just need to update the Sketch file. This could be as easy as tweaking a screen or adding a new one or removing one. It takes minutes to do and can be done in parallel with actual development. If you decide to do parallel design and dev be sure that the developers are working on things you won't be changing. Example: You have a map UI and know for sure that the store component won't change but the layout around it will. Your dev team can work on that store component without fear as it's not part of the test. As a product manager this helps you deliver value sooner rather than later.
This method is the most expensive because you generally have to have something tangible for users to test. You've already gone through design and some development. I recommend going for the MVP approach. You don't need to cover every edge case but you should have the happy path completed in some form for users to give feedback on.
For our onsite kiosk tests we offer to buy users their lunch. This does have the risk of users giving us biased feedback but we reduce that by watching them use the product/feature. We go into observation mode. One team member will give a scenario and answer questions while the other will observe and take notes. We really focus on watching what they do more so than the verbal feedback they give us.
BETA User Tests Groups
If you are fortunate enough to have a soft rollout you can work in a small BETA. Most brands have a group of super fans who love to interact with them. If you work for one of these, you can canvas your user base for volunteers. Feedback here may be biased, but any feedback is welcome if you are in a pinch.
If you're all in, you're all in. A simple and effective way of getting real user feedback is a phased rollout. You can do this with either a feature flag or the app stores. You definitely increase risk but you control the impact. If you see a spike in complaints or failures you can take immediate action to remediate before moving forward.
Implement a Design Process
This is a longer term step but a design process is very important for making quick decisions. We've been using the a variation of the Design Thinking Process. This is a critical piece of our workflow. It helps us iterate through ideas quickly to land at solutions faster.
The gist of it is that you start with a proposed solution and end up with something that has been battled tested by users. The solution you start with is rarely the one you end up with after testing with users. You'll discover gaps and new requirements that you wouldn't have without the process. By doing this you'll end up with a deeper understanding of your users and their needs. We won't go into a full explanation but you can read more about it here.
I don't recommend pinching ideas from competitors... but I am definitely not above it. If you find a solution that you think solves your problem, study the approach. Often times competitors are trying to solve the same problems and rarely do their solutions stray from yours. After all, you are competing for the same consumer and generally solutions don't stray too far (example: loyalty program are moving towards points vs spend or visits).
In a perfect world you have enough time and money to make the best decisions possible for users. Unfortunately, that's not the reality most of us live in. We can, however, minimize the risk by taking positive actions when placed in undesirable situations and in some cases can work proactively to reduce the risk.
I can't imagine delivering a feature that we didn't test. I say that -- but I am 100% guilty of doing it. Recently even. I regret it but I did it because we had to. We were hemorrhaging money and the solution was an updated experience. We launched it and it worked, well it solved the problem -- it brought down our billing significantly. Huge win. I'll write a case study on this soon and link to it from here.
Thank you for reading.
Walter Colindres is an experienced Product Manager, UX Designer and Developer. If you have comments or questions, you can reach him by email.
- Remember, your stakeholder is, in most cases, not your target audience. You need their approval but the success of your solution depends on actual users. ↩
- From experience there are numerous things at play: politics, sales dip, pressure from higher ups, marketing campaign dates, misconceptions around what testing is and so on. A lot of these are out of control so let's focus on something that is: testing the experience. ↩