What kind of automated testing should a startup have from the beginning through the first six months of live operation?

Practically speaking, it will be hard to develop test automation while iterating. Especially since product features are still being defined and you’re likely to not only change the UI and API, but pivot completely on what you do.However, TDD will help developers think more clearly about *what* they’re developing. It will make coding faster, because they will develop fewer features.YAGNI is the quick win for TDD. The longer term win, refactoring, probably isn’t going to make the cut. TDD will also help your design.So I’d suggest avoiding system testing, especially UI driven test automation. I’d also suggest limiting the amount of unit tests because a lot of your code will be throw away.That doesn’t mean don’t use unit tests for product exploration, algorithm verification, etc. Any time you drop to a debugger or start sprinkling prints into your production code you should probably write it as a test, even if that test eventually gets thrown away. It’s easier to delete a file than to delete a single line (e.g. print “debug output”) from a file.BDD is a huge win — even if you don’t automate. Because succinct, codified “plain text” documentation of features helps everyone to understand. And it drives to the core of what the domain knowledge is.But, the biggest win you can possibly get with automation, is to set up continuous delivery. Having an build/deploy script that runs at least every day (even if it’s just someone running it manually) will make sure you don’t end up with insurmountable technical debt.And the most critical thing you can do when developing quickly, is to be able to demo new features for investors/customers.

Given the many and well thought out answers above, I'm starting to get the impression that the true answer to this question is "it depends." It depends on the complexity of the software at the stage of minimal viable product (MVP). If your MVP is simple, by all means keep testing and automation of testing to a minimum. However, if your software is sufficiently complex that a lack of tests will slow you down before you reach MVP, then you should certainly be writing tests. That being said I'm only making my forays into software development now as the "business guy" on my first startup and I'm also the person who asked this question, so take this answer with a grain of salt.

Avoid waste. If you are in a startup position first understand your business model and gauge the level of investment you wish to make and off set that against the business opportunity. I recently heard of a small startup implementing the whole automated delivery chain to level that is far in excess of their need. This is waste. Implement the right amount of effort to get the business going, this means the minium amount of coding, testing, design etc etc etc

You might be doing it wrong if you're testing that you're doing something correctly before you've tested that it's worth doing.

The approach really depends on the kind of the product that's under making. A.  I have worked on products whose customer base is very specific and in these cases what matters is to get the work done, have first few feedback cycles and see how the product is being recieved or does it needs some tweaks to boost its usability. Automation in these cases is beneficial and comes handy only after a product a gone through 1-2 initial versions.... this helps the testers to understand the scope and design automation framework accordingly. B.  While there are cases where the product has to deal with 100s of end customers on the day of launch...I specifically mean products in e-commerce and mobile app space.  In these cases It always proves cost effective and beneficial to have test automation to be brought up in first 6 months of the start-up itself.  While UI automation will be an imp part here but to prepare a framework to test functionality + UI across platforms....Automation is the right strategy to get into. Bringing up automation framework along with development framework enhances collabration between teams and promotes TDD as well. Output is a better quality code with thorough testing strategy in lesser time to production.

Your product velocity should be greater than your product robustness. If your service is offline for 36 hours and no one gives a shit, the problem isn't the availability - it's the lack of users caring. There is a balance though. Deploying software that looks like ass, feels like ass, and crashes constantly won't exactly make for a good first user experience. I believe the place you want to be is that "it works" but if you get 10,000 users it'll probably fall over. Products that are built to scale before they scale are probably the most depressing software engineering endeavors.

Good level of automated functional and unit testing should be part every project / feature. Writing an automated test before writing code is sometimes a valid practice (TDD). Also, sometimes code should be written in such way to improve testability (simulate interaction with certain hard to simulate modules). Why? It is much easier to iterate and play with features (and start looking for product/market fit) if you have everything already automated.

A startup has to get the product out the door in its maket. If it floats, then it will make cash. And then, you'll be able to improve it. Of course, if you are playing with a ton of other's people money, you can stuff in tests and get better quality. But if it is your own money, you'll tend to see things differently. Perfection is the killer of success. Remember that when you'll be drowning in tests. Make architectural spikes that deliver and test the key components. Edge cases will occcur and make things break. So think about a fast recovery mechanism and keep the data safe at all times. This will beat testing when the shit will hit the fan (which is about sure if your product is going to be successful).

I believe that the critical question is what happens if the software breaks?  Do we lose $10B in 30 seconds or do we just stop making $1 for 30 seconds.  The answer to that question is a crucial lever in how we should determine what risk we should take on speed to market versus reliability to market.   What bad happens if it breaks?  Well, we stop making money while we fix it and we get some degree of distraction from sprint tasks.  I’m not so concerned about the first, but definitely concerned about the second.  If it’s a minor distraction, that may be okay.  If it’s daily and time consuming, then that’s not okay and it’s probably time to write some tests.  On the other hand if it’s once a month and we just restart the app, that may be okay to live with if we get significant leverage by getting the product into market sooner.  Be wary of constant production distractions as we need predictability in our velocity as we have a lot of downstream dependencies on our ability to deliver what we promised to deliver on time.     This should be a healthy discussion as part of every sprint.  We should have conversations on time-to-market versus risk to market.  It is best in a sprint estimation process for someone to be on each side of the conversation. We should have a speed advocate and a safety advocate.  If we have everyone on one side or the other and no discussion then we are not doing our jobs properly.  We’re either gold plating every deliverable or we’re throwing garbage out into production which is going to bite us.  The answer is not in either extreme.  Every case is different and the team should discuss the trade-offs and reach a quick consensus based upon their experiences.  Teams will get better at this over time.