Experience Series: Product

17 min readJun 11, 2019

This post is part of my Experience Series where I share opinions I have formed from building companies and running product. Each posting is a break down of a company function. This post focuses on product specifically.

Ideation

Come up with an MVP and socialize it. Hint, most of the time this does not involve writing any code. The key word in this process is “minimum”. What’s the least amount of work you can do to test responses.

As a developer, it’s taken me years to apply this sort of advice and even today I struggle. Using tools like Balsamiq mock-ups has given me a way to “build” without actually dedicating the effort. It’s a good middle-ground for moving beyond the MVP.
At PassiveTotal/RiskIQ, I made a number of mistakes by skipping the MVP process. In some cases, I got lucky and the feature was used. Other times, it was a complete flop. This obviously wasted time, but also hurt morale for all the teams involved.

Use tools or a whiteboard to mock up your ideas and get feedback. As you mature the idea, begin building better mockups. At the end, you should have a pretty good idea of what you will want to produce and how it will function.

I work from home and have a whiteboard in my office. I tend to sketch ideas quickly, socialize them to peers and get feedback as fast as I can. Once I have baked the whiteboard mocks enough, I move over to something more formal like Balsamiq Mockups. A “mock” to me is not just wires, it’s what I want the end state to ideally be.
For the longest time, I thought requirements were enough to capture what I wanted to build. I’d hand these over to engineering and end up with wildly different results than what I documented. The issue––absent a visual, assumptions will get made, even with the best requirements. Consider the following description, “Something you can pick-up, read and take with you”. This could be a Kindle, Newspaper, Magazine or a cell phone. Paired with a visual, there’s less room for assumption and a higher chance I will end up with what I want.

Test your theory, survey potential customers, get harsh feedback and ask open-ended questions. Avoid the self-assuring garbage of “will you buy this”, or “is this a good idea”; those usually get answered with a “yes” and are no help to your approach.

Early on with PassiveTotal, we asked the self-assuring questions because we didn’t know any better. You are going to feel like an imposter and that’s okay. Get the assurances when you need them, but don’t let it drive your ideation process.
For surveys, I use Google Forms and it works great. I try to keep the amount of questions to 5 and always provide an “other” option if I want alternative views to what I am asking. There are times when I will embed surveys into my marketing campaigns, especially with emails I send for those who stopped using the product. These are rarely filled out, but can be helpful.
When you have an idea you’re excited about, it can be challenging to ask open-ended questions and not “lead the witness”. I like to pepper the person with questions and let them know I will explain my ideas after that process is complete. It gives them incentive to answer honestly and they have no conceived notions.
For open-ended questions, walk back from your “solution” and ask more about a user’s workflow or challenges. Be sure to practice the process because it’s not likely to feel natural right away.

Know your audience well. Who are they, what do they do, what do they care about, who manages them, who do those people report into, how do you engage them, etc. Your audience shapes your idea and approach. Build out personas for your major users.

For PassiveTotal and NinjaJobs, I got lucky in that I was the developer and the audience. Even then, I could have benefitted by sitting down and mapping out the persona in detail. Here’s an example of the type of personas I build today.
I’ve published my general “security organization” model I use when thinking about personas, their roles, which department they are part of and how they fit into the larger ecosystem.

If your idea is good and you keep talking about it, someone else will implement it. Talk to a trusted individual, your dog, your spouse or a rock if you have to, but don’t go telling the whole world what you intend to do unless you’ve done it.

This sounds like commonsense, but when you’re excited, it’s hard not to want to tell others about what you’re doing. It’s okay to share some detail, but recognize that ideas are hard to protect.
If your idea turns into a company or product, you are going to need someone to chat with about details or challenges. Having trusted contacts who you can be open with is going to drastically improve your day-to-day.

Product market fit in cybersecurity is tough to work out. Don’t freak out if you are unsure where your idea lands. Validate, do your best and start iterating.

Absent true product market fit, just look for competition and that may help validate your idea and give you some sense of direction.
Try not to get sucked into the industry “flavor-of-the-week”. There’s always some marketing hype on a feature or type of process. I personally consider these trends to be “noise”, but it’s important to recognize them and extract why they resonate with the industry.

If you are building a business around your idea, it ought to come with a vision you can communicate out to prospective clients. If you lack this or aren’t passionate about your idea, you may not want to start a business.

There’s been countless times when I really liked an idea, but didn’t love it. I was able to keep myself motivated for a few weeks or months, but as I started to meet resistance in the process, I felt less inclined to push through.

As long as your idea or product is free, there will be people who use it or abuse it. Those same people may tell you to start a business and that they would totally pay for the service. Expect an immediate change once you start charging for the product and don’t take it personally.

I am huge fan of freemium models and try to work those into all my products. My view is that everyone should get the benefit of my work in some capacity, even if they don’t have budget for it. These models aren’t for everyone and come with a lot of baggage, so do your best to game-plan out how you would go to market.
On immediate changes post-paid-model: you may have less engagement, users will complain, you will be held to a new standard, certain features will be expected, you will begin getting inbound requests for information, competitors will start paying attention to you, etc. It’s a simple step forward that has some big impacts to your idea.
With NinjaJobs, I felt like we stayed free for far too long. There was no direct impact to the business by waiting, but indirectly, we could have captured revenue much faster. One of the nice models of partial free though is that it solves the “marketplace problem” in which we need to have a constant stream of jobs being posted in order to keep seekers engaged.

Roadmaps

Early on, roadmaps are less important. Your vision can help fill in gaps, but in general, your first invested users or customers will help drive a lot of your features. Make them happy, keep true to your core and constantly retest your approach based on new data.

If you work with your first customers, be really careful not to end up building a product that only caters to them. Take your progress and socialize it to others in your network, even if they aren’t customers.

There is no set format to a roadmap, though it generally consists of quarters (time) and a few key initiatives that may or may not have a single or set of themes.

I’ve done a number of these presentations and even when the customer loves your product, I don’t find interactions to be terribly high. In other words, don’t kill yourself in trying to plan out your next 5 years. Focus on the next year and have a vision that reaches beyond.
AHA has some good guidance on formats and creating a roadmap. I’ve used their product in the past and thought it was nice, but ironically, too much of a “kitchen sink” for me. That said, their advice is sound and very helpful.

A good roadmap/vision can help you land you a deal, even if you are still very early on. Depending on the customer, they could see you as an investment in solving a more specific problem they have.

As you mature, roadmaps are expected when speaking to customers and investors. Keep it high-level, tell a story and sell the vision.

Execution

Building a tool is not the same thing as building a product. If you are trying to build a company, you need to balance what makes sense for the business versus appeasing a technical audience and gaining accolades from your peers.

With PassiveTotal, we spent a lot of time building features for our power-user base and totally skipped out on considering those who were less technical. As we grew, we noticed that a larger percentage of our users would drop-off in the product. This feedback forced us to up-level the product and begin educating the market more.
Power-users are likely to be a small percentage of your real user base. Listen to them, but don’t let them completely control your roadmap. At some point, their feedback stops being helpful and it starts to harm your prospects of catering to a larger audience.

If you are able to build your product while working a day job (moonlighting), strongly consider that. Yes, you’re working two jobs, but you have far less risk which means far less stress. Working two jobs ends up being an early preview of working a startup anyway.

I developed PassiveTotal in between finding jobs and later sold it with my co-founder while working at Facebook. During our first year, I only went two months without a salary.
NinjaJobs was developed over the summer while PassiveTotal was being sold. I don’t know how I managed to do this, but it involved a lot of nights and weekends.
All my other coding product or tool projects were developed in between jobs or during them. If you decide to go the route of “moonlighting”, just be sure to check your employee agreement for who owns intellectual property you develop.

As exciting as it is to deliver new features or updates, don’t do it at the cost of your users. Test your builds and do your absolute best not to break anything.

I used to release changes to PassiveTotal and NinjaJobs at-will. While I didn’t break the platform often, there were times when I would introduce bugs, or cause a poor user experience. These days, I wait until the feature has been tested and even then, will delay the release until it makes sense.

Do NOT: Release on a Friday or the weekend, do a big release prior to a vacation, release without testing in a staged environment.

It’s tempting to think nothing bad will happen and usually it doesn’t, but when it does, it will always be during one of these times. Best to avoid them completely and release when you are going to be around a few days.

Early on, you have the advantage of moving fast and implementing quick fixes or ideas. If done properly and in tandem with a customer, they will forever be loyal to you.

We have PassiveTotal customers to this day that stay with us because of our early work with them.
There are occasions where I need to call up a customer with NinjaJobs and listen to their feedback. In rare cases, I’ve been able to deliver a change while on the call. There’s no greater feeling than hearing the feedback from a delivery like that. People think you are a wizard.
Balance the moving fast with the advice above on controlled releases.

As you release, continue to find ways to test or survey your audience. Ensure you are still moving down the right path and that what you delivered provided value.

At RiskIQ, we try to assign a set of customers to each of our major features, so we can socialize the gains we make along the way. As we get closer to release, we organize to get final tests and feedback worked out. An added benefit here is that some of these customers will provide us quotes or materials we can later use within our product marketing.

Keep yourself and your team on track as best as possible. With one or two people, this is easy, but as you grow, it becomes difficult. Get in the habit of having healthy process that maintains a consistent flow.

At RiskIQ, we do this through a bi-weekly sprint cycle and regular meetings with varying parts of the organization. Even with all the communication, we still run into issues, but everyone does their best to keep up.
NinjaJobs is still small and Slack is sufficient for our communications. Rarely do we need to meet-up in person, but will do that just to hang out and exchange ideas.

Prior to a release, ensure you have your materials all ready to go. This should include release notes, value points, any differentiating factors, who (persona) benefits from the feature, how you want to socialize it and any documentation for users.

Formally called Product Marketing, this is really tough to get right if you don’t have a dedicated resource. If you are having to juggle multiple initiatives, you need someone who is highly process-oriented. In RiskIQ, I have the Product Managers put together the larger material items and the Product Marketing team use that as the foundation for their material development.
What makes this process difficult is largely the timing. There’s so many elements to a feature release that need to be coordinated. With RiskIQ, we have yet to find a perfect solution, but regular cadence meetings and milestone checks help a lot.

Maintain forward progress in your development. If you are doing an alpha or beta, pick some goals and a timeframe. Don’t let it run forever or keep increasing the scope.

At RiskIQ, we have multiple products with multiple initiatives and multiple customer concerns. When we release betas, we need to be sure to institute an end date, otherwise other priorities can slip in and distract the team from revisiting the work. What works well for us is baking these dates and approaches into our larger release plan. If there is a delay in any of the process, we shift to the right and continue our progress.

There are a ton of sources to monitor to help iterate on your idea. Use a combination of your subject matter expertise, market analysis, customer feedback, user feedback and competition as guides.

It took me at least a year to realize that this is largely what Product does all day––a combination of “all the things”. You have to be able to context switch and keep track of various schedules, otherwise you’ll never make any true progress. This can sound intimidating, but it’s honestly why I love the position. Every day is slightly different and there’s no shortage of items I could work on.

Onboarding

Define the state of your product from day zero all the way to day 365. Ensure that your product changes to support the needs of the user. Early on, focus on educating and pushing towards happy paths. Later, focus on value hooks that keep them coming back.

When planning out your features, it’s important to account this concept of state. Generally, I find most people think about the feature and how it would ideally manifest itself in their product, but don’t put as much attention on the first experience or the 500th experience. Several times a year, I find myself re-registering for products I’ve built just to see if there’s any new lessons I could apply.

Study onboarding processes of other companies and products you like and then consider implementing them.

I found UserOnboard to be a helpful resource when thinking about onboarding. Their package of training materials helped augment how I think and the tear-downs of popular products give me broader ideas I could apply to my own products.

The more friction you place in adopting your product, the more you are likely to fail. Make it easy for users to sign-up, and begin using what you’ve produced.

For any of the products I develop, I like to keep the initial registration very simple––email and password. Once the individual goes to confirm their email address, I know they have invested some effort. It’s during the setup phase that I will collect additional metadata to describe the user and understand their interests.
With NinjaJobs, we don’t bother to validate email addresses because it’s not important. Someone applying for a job has a vested interest in having a proper contact email and accurate name. One could argue it would still be worthwhile to validate, but the lack of including an additional step means more users are likely to sign-up and apply for a position.

Try to forget everything you know each month in an effort to view your product as an everyday-user. Abstract technical concepts, so users aren’t overwhelmed or turned-off by your approach.

When I first built PassiveTotal, it was aimed at power-users. We were taking technical data and consolidating it within a single platform. Over time, as we grew, so did our user base; it became more diverse and introduced users who had a lack of technical knowledge. As we spoke to these users, we realized our product was not very helpful to them. At that point, we had to forget what we knew and redesign elements of the product to ensure even those with limited technical knowledge could use it.
With Blockade.io, the user base is largely non-technical users. In building that product, it was important that I make using it as natural as possible. In other words, I didn’t want someone to have to change their browsing habits in order to get the security benefit I was offering. This was a fun exercise and one I ultimately felt I succeeded in achieving. Getting to that point was a long process though and required many iterations on the product before I got it right.

Onboarding is best done in well-defined stages. Build a view into these stages to know where people are running into issues. Are they not validating their email addresses? Are they filling in a form, but not submitting?

Having stages is great as it gives you insight on how you may want to improve your onboarding. When we first started with PassiveTotal, we sent emails from our single server and didn’t bother going with a transaction mail provider like Mailchimp. This worked for a while, but eventually we got flagged for spam. How did we know that? We saw an increase in users who would register, yet not confirm their email. We move to a proper provider and solved the problem, but without the insight, we would have just continued to rack up unconfirmed users.

Measurement

Product Qualified Leads. Learn about them and find a way to create them within your metrics. Any major action in your product should emit an event you can measure and inspect.

I prefer to go with the notion of a PQL due to the freemium models I typically employ inside my products. By design, not all of my users are going to upgrade to a paid version. A PQL model ensures that I separate the likely opportunities from those who will remain free forever.
There are times when you will run into a power user who will never have budget due to their circumstances. PQLs can help surface these users too and it gives you a chance to extend an alternative offer. With PassiveTotal, if we find a strong user who lacks budget, but is willing to publish research, noting the use of our tool, we will consider giving them a research account to use.

As best as possible, be able to tell the story of your users through data, without their direct involvement. Even when your product is good, it’s hard to get people on the phone and giving real feedback. No use is feedback — you provide no value.

In my view, this is one of the most important things you can do in building a product. Before I begin development, I will outline every action I would expect a user to take. For example, “user register”, “user confirm email”, “user login”, “user logout”, “user update settings”, etc. Every action has a baseline level of context (user who executed, date-time, location, url, etc.) and an action-specific level of context. Anytime an action is taken, an emission takes place to capture this data and store it locally and/or within a analytics system. This instrumentation allows me to truly understand how my products are being used and where I should consider focusing my efforts. The more time you spend with this sort of data, the better you are at telling a story and the more efficient you can make you real user interactions.
Assuming you get someone on the phone, they may not give you the feedback you are looking for. Keeping in mind all the measurement above, it’s not uncommon to interview a user who has seldom used your product only to find them saying how much they love it or how they use it all the time. It’s not your place to call them out, but to know this beforehand and tailor your interview to figure out why they don’t use the product that often.
Sometimes data is not enough to tell the full story of a user action. For example, in NinjaJobs, I can see people search and apply for jobs, but I don’t know their exact process of interacting with a post. To visualize user actions, I like to use Hotjar. Using that technology, I was able to review roughly 20 sessions of users searching for jobs in NinjaJobs. Much to my surprise, our job search process was really difficult to use. At the time, we paired a query and location into one field. The result was users only being able to choose one or the other. Seeing this in the videos made me realize that we needed to split the input search field into query and location, a simple change that would go on to make a massive impact. After implementing the change, I saw the number of searches drastically reduce and become far more focused.

Don’t shield yourself from the numbers. Whether it’s statsd or some 3rd-party tool, look at your numbers and be honest with yourself. If the feature you just released was a flop, don’t bask in the negativity, learn where you went wrong.

To measure engagement and platform usage in NinjaJobs, we use Datadog. We formally used that with PassiveTotal, but later replaced it with a combination of internal tracking, Newrelic and Woopra. Both products continue to use Google Analytics as a back-up analysis method that helps us tie in website information as well.
When releasing a new feature, you should have some defined success criteria that you can measure via metrics. If adding a new feature should increase conversion, pick a rough percentage goal and then measure it once you release.

Metrics you collect should be sent to a number of locations including a behavior tool where you can generate reports, a CRM for tracking deals and a marketing tool for emails or in-app notifications.

PassiveTotal and NinjaJobs used Datadog for tracking events and providing reports. PassiveTotal later switched to Woopra for behavioral tracking.
PassiveTotal initially used Hubspot CRM prior to our acquisition and it was hands-down one of the reasons why we were able to exit how we did. After joining with RiskIQ, we switched over to Salesforce.
PassiveTotal and NinjaJobs leveraged Mailchimp and Mandrill for sending emails to users. Both platforms continue to use these tools today, though PassiveTotal also makes use of Marketo.
All my products have used Google Tag Manager, Webmaster tools and Google Analytics for generic website, and web application tracking. Protip, Google Analytics will allow you to emit “events” and measure those over time, also allowing you to associate that data to goals.
NinjaJobs and PassiveTotal both use Intercom for in-app messaging. NinjaJobs use of Intercom is more automated deal conversion whereas in PassiveTotal, we primarily leverage Intercom to share updates and perform support requests.

Identify the ideal or “happy path” within your product and ensure you have a way to measure it end-to-end. If users sign-up and never hit the path, try to identify why. Call them up, message them, do whatever it takes to learn.

Mentioned above, but worth pointing out again. If you can’t get ahold of your users in any reasonable way, consider using a technology like Hotjar to have the sessions recorded. Data is scrubbed prior to being sent to Hotjar, leaving you with just the interactions of your product.