Experience Series: Engineering

https://dilbert.com/strip/1995-01-20

This post is part of my Experience Series where I share opinions I have formed from building companies and running product. Each posting is a break down of a company function. This post focuses on engineering specifically.

General Opinions

The language you write your product in doesn’t really matter, as long as it solves the problem you are trying to market.

• PassiveTotal and NinjaJobs were originally written in Python. Is it a highly scalable language for dealing with web-based problems? No, but it got the job done and continue(s/ed) to do so while making money.
• Be mindful that you may not be the only person to eventually support your product, and so whatever language you do choose, make sure it’s at least known to others.

First-pass code — even with the best intentions — will have issues and likely need a major refactor or rewrite. Focus on delivery, not scaling (can’t scale something not delivered).

• For me, engineering is a never-ending discipline. As I learn more and read better code, naturally, my code starts to improve. Even when I felt like I had successfully designed the perfect system, I’d view the code 6-months later and find glaring issues.
• With PassiveTotal, we undertook several rewrites to bring the system front-end and back-end to match the technology stack of RiskIQ. Ideally, we’d avoid such rewrites, but as part of scaling, we needed our core staff to be experts in the system.
• Specific components of PassiveTotal eventually failed to scale as we grew the business. While this was stressful to deal with, it was also a validation that what we had put together was getting used by our customers.
• In initially designing NinjaJobs, technology was chosen that best suited the job. There was no intense discussions around scale or longevity, yet 4 years later, the original system continues to thrive and work as intended.

Stay focused and avoid the engineering pitfall of “building all the things”. As a startup, it’s okay to be cheap, but your time is money. If you need to manage contacts and sales pipeline, don’t build your own CRM, go and pay for the service.

• Most of the systems we paid for in PassiveTotal and NinjaJobs were ones tangentially related to the business––customer support, marketing, contact management, sales operations, accounting, devops, etc. Paying the small fee––or sometimes none at all––let us focus on what was most important, building the product.
• Many existing packaged solutions are not going to meet 100% of your needs and that’s okay. That’s not a reason to go build your own. Identify the gaps in the outsourced services you use and if it’s important enough, get creative.

If given an MVP to implement, don’t go hog wild on the implementation. The key to an MVP is the metrics used for measuring. Get as much coverage as you can and move on.

• As a developer and someone who enjoys writing code, this is easy advice to say and the hardest to put in practice. When you think about it, most MVPs could be a button, that when clicked, logs a message-of-interest and kindly thanks the user for their input. Seldom do you need to engineer much of anything for an MVP. Always validate before you build!
• So many of the early PassiveTotal features were driven from my personal experience doing analyst work. While they were great ideas, they often failed to get the level of engagement expected when delivered. Had I validated the ideas properly, I would have saved myself a lot of time.
• Prior to my involvement in NinjaJobs, the product was literally an email digest and a single PHP file––talk about an MVP. Not only was the company placing hundreds of people in jobs, but it was also making a hefty profit. Moving it over to a platform later quadrupled the traffic, but doing so came at a risk-free cost since so much validation was done using the MVP.

As best as possible, build feature-flagging or gating into your product. You can roll your own or use a service. It’s a great way to deploy rapidly and release selectively.

• Working at Facebook and RiskIQ exposed me to the power of flagging within the product lines. There’s a beauty in being able to release constantly to production, yet never break the experience for your users as the functionality exists behind a flag. Depending on your instrumentation, you can selectively release to certain users and truly control your roll-out.
• Products I build today contain some element of flagging, though it’s not usually as granular as a company like Facebook. For my projects, “groups” tend to serve a good balance for who should see a certain feature and when. Unlike your traditional RBAC groups, my groups are typically organized around a feature or beta functionality. It’s far from perfect, but it works and is not terribly difficult to implement.

If you plan to deliver an API, put some serious effort into the documentation and language-specific abstracts. My general approach is that a non-developer should be able to use one of my APIs and be successful without writing a single line of code.

• There are plenty of systems to take code written and output an API client. While these tools save you time, they often deliver a poor experience to the end user. APIs should be written for humans, by humans and not for humans, by machines.
• As the PassiveTotal API evolved, we found ourselves working with an audience who could cobble together scripts, but were far from developers. Giving those users a copy/paste example gave those scripters confidence to use our tools.
• With Backscatter (RIP), I went a next step in making the tool approachable by providing multiple (curl, bash, python, library) code samples for each call. No matter how the user wished to engage with the product, they could find a copy/paste example.

Don’t release experimental APIs to the world unless they are clearly marked alpha and subject to change. Once someone implements your API, it’s nearly impossible to get them to change to something new, even when there is new content. It’s best to plan your design, test it, actually have someone implement it and then release.

• To this day, decisions we made early on within the PassiveTotal API continue to haunt us. Could we deprecate the functionality and move on? Sure, but we have to be mindful of our audience. Releasing a new API with little new functionality or changes is not a good way to get users to upgrade.
• As a test case for your API, have a developer (customer) build a few tools or scripts with your alpha release and listen to their feedback. What may look good in documentation, could pose annoying in the real-world.
• Libraries are great ways to abstract away from your users directly calling your API. These are also helpful when integrating with other products. If done correctly, even if you need to change the API later, you can just update the client to reflect the changes, leaving the end-user none the wiser.

The goal early on is to build a product, not an engineering masterpiece. Your implementations do not need to be perfect. If the code works, handles under the load you have and meets customer expectations, you are doing it right. That said, don’t build a house of cards.

• PassiveTotal was largely a successful proof-of-concept when sold to RiskIQ. NinjaJobs was making healthy profits as a single PHP script and email digest. Blockade.io was formed using all free services from AWS and supports hundreds of users. Focus on the product, not the technology.

Use technology you know and are comfortable with. Save the new tech or experimental ideas for side projects. Test them there, find the issues and then implement if you think it’s worth it.

• Running PassiveTotal and NinjaJobs at the same time afforded me the ability to run experiments between each company. I often used NinjaJobs as a testing ground where I would roll-out new features or technology in order to see how customers interacted with them. If I saw a lot of success, I would understand why and identify if that same success could be brought to PassiveTotal.
• Today, I write code on my own time and poke around at different services in the market today. It’s a fun way to keep my skills fresh and allows me to gain exposure to technology I may want to use in my future products.

As you begin to scale, consider technology that a greater audience knows. If you need to hire someone later on, it’s best to have a large talent pool to choose from. Mature/widely used technologies also tend to have better support overall.

If you have a team writing code and you’re in product, remove your write-access if you can’t follow process. Your code slinging days are not done, but you need to respect your team and not undermine their efforts.

• This assumes you are moving into more of a product role. I struggled for a long time to let go of writing code when I joined RiskIQ. It’s not that my contributions weren’t valuable, but I was not following process. There was a number of times that my code would introduce a bug, cause a distraction or simply fail to show success. If you’re going to write code, make sure you are willing to do all that comes along with it, including support and maintenance.

Leverage frameworks to speed up development, but don’t forget it’s possible to manually instrument. Sure, you can push a full MVC pattern into your user’s browsers, but do you really need it? Would HTML and a few JS calls be sufficient? Think of the fragility and support over time.

• Take this point with a grain of salt, as my bias for tried and true methods is showing here. I’ve personally had poor experiences with modern Javascript frameworks that aim to handle immense amount of application logic on the front-end. Arguments I often here in favor of these frameworks don’t usually account for the long-tail support of a product overtime.
• All of my projects have been developed largely using good old fashion HTML and Javascript when I felt it was useful to improving the user experience.

Fast-track your application look and feel with a paid template. Buy it and extend it to make it your own. You’ll save a lot of time focusing on real product development, not the bones of your application.

• Most engineers are not designers. I myself am not a designer, but can produce decent looking products. For every single application I build, I start with a template––typically from wrapbootstrap––and adjust it to meet my needs. I can’t stress how much time this process saves me. Plus, as that template is updated, you can incorporate those into your project.

If you write several applications, consider making your own bootstrap framework in the language of your choice. This would include the basics like user login, registration, session management, administrator controls, logging, etc. These save you time in the future and creates a consistent codebase to operate from.

• I am most comfortable using Python, and so my boilerplate code is written in that language. When I want to start a new project, I copy a generic base that covers 80% of my needs and then build custom logic within it. Using this approach, there are times when I can go from idea to deployed application within less than 24 hours.
• This same approach can extend to the MVP phase of a project too. For “coming soon” pages that collect email addresses, I use a number of free-tier AWS services (Route53, SES, S3, CloudFront, API Gateway) and LetsEncrypt in order to stand-up a professional page that I can use for promotional materials.