Scientific Learning

In my previous post, I wrote about Lean Startup. And I found that it is very interesting that I have come through these things.

  • S.M.A.R.T (specific, measurable, actionable, realistic, time-bound) (post)
  • Machine Learning – Deep Learning – Convolutional Neural Network (CNN)
  • Statistics
  • Kaizen (改善法) (post)
  • Agile development. Test development development (TDD)
  • Research study

When I read The Lean Startup (Ries, 2011), I learn about lean manufacturing. Then only I discovered that kaizen is also part of the lean manufacturing. Wow! I realise that, all of these things are related. And the core of the concept is, “scientific learning”.

In Lean Startup, in my opinion, the fundamental concept is Build-Measure-Learn loop. According to Ries (2011), the “Entrepreneurship is a management, learning, and involves experiments”. Throughout the whole entrepreneurship, it is about the learning. It is a scientific learning, because entrepreneurs need to form hypotheses and then test the hypotheses with experiments. Furthermore, the method can be replicated,  The results can be measured and validated. It is a validated learning. The main purpose of Lean Startup is to make the Build-Measure-Learn loop faster, then for each iteration, the product will be optimized and improved through the learning.

This is exactly how Machine Learning works. In Machine Learning, the core concept is the gradient descent. For each calculation of output (≈ build), the machine get the errors (difference of actual output and the desired output) (≈ measure). Then using the errors, machine will update the weight so that it will produce the actual output that is closer to the desired output (≈ learn). It involves small batches and many iterations throughout the training.

Agile development shares many characteristics with Lean Startup. It involves iterative development and daily stand-up meetings (scrum) which improves the communication. Besides that, TDD is also part of the agile development. Why is TDD important? I have wrote about it previously. But after reading The Lean Startup, I found that TDD is more than just making the product stable. As we use TDD, we can test the features or functions with command-line. It is extremely faster than testing the functionality by running the application from beginning to the end manually. Imagine that you want to test a webapp functionality, you have to open the web browser, login, click-and-click-and-click, refresh-refresh-and-refresh. If using TDD, as lean manufacturing, this can minimize the waste drastically. Similarly, you write the code (≈ build), you test it (≈ measure), and you make changes to optimize your code (≈ learn).

Unluckily, when I was doing PhD research study, I didn’t know about Build-Measure-Learn. Research study also shares a characteristic of entrepreneurship, i.e. uncertainty. However, getting supervisors’ feedback is not an easy job, as most of the time our supervisors are hardly to give instant response. As a result, unlike entrepreneurship, research study has difficulty to accelerate the Build-Measure-Learn loop. And most of the time, research study requires enormous prior knowledge of the research area.  Without fulfilling this requirement, supervisors’ feedback doesn’t help much at all.

When I was a lecturer, my institution adopts OBE (Outcome-Based Education). It shares the characteristics as Lean Startup and S.M.A.R.T, as the outcomes have to be measurable and observable. However, I personally don’t agree the ways of implementation of OBE, because it makes the teaching become very rigid, especially setting up the exam papers have to fulfil the criteria strictly. I agree that the outcome should be measurable, yet the ways of implementation is exhaustive and not really effective. It is muda and muri. But I believe that, if students are able to implement Build-Measure-Learn loop in learning, the learning will become effective. For example, “build” the knowledge through lecture class, group discussion, and other activities. “Measure” their gained knowledge through various tests. “Learn” from the results of the tests. And, repeat. Each iteration should be small. By each iteration, the students should able to learn through the failures they made and optimize their solution to get the desired result. (Nevertheless, I believe that this implementation is impractical, due to the cultural pressure and stereotype that the students results should be a bell-curve. W*F!)

As a conclusion, by using the scientific learning method like Lean Startup, kaizen, or even Machine Learning in our daily life (small batches, iterative, and measurable), surely this will make our life better.

After reading The Lean Startup

Because I was planning to start a small project, I have developed a prototype. And I asked a friend for some advice and showed the project to him. Then he gave me his opinions, and asked me to read The Lean Startup (Eric Ries, 2011). So, I stopped my project, and start reading the book. This post is about what I get from the book.

(I will write another post for what I have learnt so far.)

Lean Startup

Entrepreneurship is a management, learning, and involves experiments.


  1. Build an organization that can test the assumptions systematically.
  2. Perform the rigorous testing without losing sight of the company’s overall vision.


  1. Entrepreneurs everywhere
  2. Entrepreneurship is management
  3. Validated learning
  4. Build-measure-learn
  5. Innovation accounting. Accounting here refers to accountability.

Vision -> Strategy -> Product

Startup has an engine, namely engine of growth. So, the entrepreneurship is to make the business grow.
Because it is a build-measure-learn model, just like driving to the destination, we need to know whether to pivot for a sharp turn or persevere with the current path.
So, we can optimize our product at the end.


Value hypothesis and Growth hypothesis

“Unfortunately, because customers don’t really know what they want, it’s easy for these entrepreneurs to delude themselves that they are on the right path.”

Minimum viable product (MVP)

Should communicate with the targeted first adopters, then only can define our MVP.

Actionable metrics vs vanity metrics

Genchi gembutsu – “go and see for yourself”

Customer archetype

A brief document that seeks to humanize the proposed target customer. It is an essential guide for product development and ensures that the daily prioritization decisions aligned with the customer to whom the company aims to appeal.

Lean UX

Need not to worry the ideas will be stolen by other companies.

  1. The managers in most companies are already overwhelmed with good ideas. Their challenge lies in prioritization and execution, and it is those challenges that give a startup hope of surviving.
  2. If competitor can outexecute a startup once the idea is known, the startup is doomed anyway.
  3. The important thing is that entrepreneur can accelerate through the Build-Measure-Learn feedback loop faster than anyone else can. If this is true, it makes no difference what the competition knows. If it is not, the startup has much bigger problems. Secrecy doesn’t fix the problems. It will be competed by fast followers.

Launch MVP under different brand name, to avoid damaging the company’s established brand

Startup is

  1. rigorously measure where it is right now, confronting the hard truths that assessment reveals
  2. devise experiments to learn how to move the real numbers closer to the ideal reflected in the business plan

Innovation accounting – Three learning milestones

  1. Use MVP to establish real data to know where the company right now is, so that you can track your progress
  2. Tune the engine from the baseline toward the ideal, using micro changes and product optimizations
  3. Pivot or persevere
    a. If pivot, needs to reestablish a new baseline and then tuning the engine
    b. Successful pivot should show that the engine-tuning activities are more productive than before


Smoke test

Measure one thing: whether customers are interested in trying a product. It is insufficient to validate an entire growth model. But it is useful to get feedback on this assumption before committing more money and other resources to the product.

When choosing among the many assumptions in a business plan, it makes sense to test the riskiest assumptions first.

Good design is one that changes customer behaviour for the better. Else, the design should be judged a failure.

Cohort analysis.

Leap-of-faith metrics.

Extreme Programming, agile development

  • Sprint (n-period iteration cycles)
  • Prioritize the work to be done by writing a series of user stories (instead of technical specification)

Split-testing, aka A/B testing

Get feedbacks through interview and survey.


stories are assigned to 4 categories: backlog, in progress, done, validating

3 A’s of metrics: actionable, accessible, auditable

  • Actionable – antidote to vanity metrics. Vanity metrics doesn’t demonstrate clear cause and effect.
  • Accessible – antidote to misuse of data, ie too many reports that cannot be understood by the employees and managers. Cohort-based reports turn the complex actions into people-based reports. “Metrics are people too”. Analytics or the data should be part of the product itself.
  • Auditable – antidote to blaming when failure. “Metrics are people, too”. Talking to customers to check if the reports contain true facts. By this, can gain insights into why customers are behaving the way the data indicate. The reports should be drawn directly from the master data, instead of intermediate system, to reduce opportunities for error.

Pivot or persevere

  • The more money, time, and creative energy that has been sunk into an idea, the harder it is to pivot.
  • Failure is a prerequisite to learning.
  • zoom-in-pivot – refocusing the product on what previously had been considered just one feature of a larger whole.
  • zoom-out-pivot – reverse of zoom-in-pivot
  • customer segment pivot – keeping the functionality of the product the same, but changing the audience focus.
  • customer need pivot
  • platform pivot
  • business architecture pivot, value capture pivot, engine of growth pivot, channel pivot, technology pivot
  • A pivot is better understood as a new strategic hypothesis that will require a new MVP to test.
  • The startup has to find ways to achieve the same amount of validated learning at lower cost or in a shorter time.

Reasons inhibit pivots

  • Vanity metrics – false conclusions
  • Unclear hypothesis – impossible to experience complete failure, without failure, there is no impetus (momentum) to perform a pivot
  • Afraid – acknowledging failure leads to low morale

Meeting. Frequency depends. Should involve both product development and business leadership teams.

  • Product development team must bring a complete report of the results of its product optimization efforts over time as well as a comparison of how those results stack up against exceptions.
  • Business leadership team should bring detailed accounts of their conversations with current and potential customers.

MVP is for the early adopters. Next level is the mainstream customers. They have different requirements and are much more demanding. They are less forgiving.

Single-piece flow (lean manufacturing)

It seems inefficient, but it has been confirmed in many studies that it is faster way to get a job done.

“Toyota Production System (TPS) is probably the most advanced system of management in the world, but even more impressive is the fact that Toyota has built the most advanced learning organization in the history.”

Sustainable growth – New customers come from the actions of the past customers.

  • Word of mouth
  • As a side effect of product usage
  • Through funded advertising
  • Through repeat purchase or use

3 kinds of engine of growth: Sticky, viral, and paid

Though a business can have multiple engines of growth at a time, it is recommended to focus on just one engine of growth.

Training programme

Standardise our work processes and prepare a curriculum of the concepts that new employees should learn.

Andon cord

“Stop production so that production never has to stop”. Brings work to a stop as soon as an unccorrectable quality problem surfaces, which forces it to be investigated. You cannot trade quality for time, else the resulting defects will slow you down later. Defects cause a lot of rework, low morale, and customer complaints

The logic of validated learning and the MVP says that we should get a product into customers’ hands ASAP and that any extra work we do beyond what is required to learn from customers is waste. However, the Build-Measure-Learn feedback loop is a continuous process. We don’t stop after one MVP but use what we have learned to get to work immediately on the next iteration.

When we go too fast, we will cause more problems. Adaptive processes force us to slow down and invest in preventing the kinds of problems that are currently wasting time.

Five Whys

Build an adaptive organization – Asking “Why?” five times to understand the root cause. This help to uncover the root problem and correct it. Using this technique is possible to reveal that a purely technical fault is actually caused by human managerial issue.

  • However, the investment (fix) should be smaller when the symptom is minor, and larger when the symptom is more painful. We don’t make large investments in prevention unless we are coping with large problems.
  • Startup teams should go through the Five Whys whenever they encounter any kind of failure, including technical faults, failures to achieve business results, or unexpected changes in customer behaviour.
  • Proportional investment

Avoid to making Five Whys into Five Blames which pointing fingers at each other and trying to decide who is at fault. To avoid this,

  • make sure that everyone affected by the problem is in the room during the analysis of the root cause
  • The meeting should include anyone who discovered or diagnosed the problem, including customer service representatives who fielded the calls, if possible.
  • should include anyone who tried to fix the symptom as well as anyone who worked on the subsystems or features involved.
  • If the problem was escalated to senior management, the decision makers who were involved in the escalation should be present as well.

If blame is inevitable, the most senior people in the room should accept this: if a mistake happens, shame on us for making it so easy to make that mistake.

Five Whys analysis is to have a systems-level view as much as possible.

(Interesting point. If a new employee can easily break the system, it is not the new employee to be blamed, but the whole team fault.)

Five Whys requirements:

  1. Be tolerant of all mistakes the first time.
  2. Never allow the same mistake to be made twice.

Don’t use Five Whys to solve “baggage” issues

  1. Use it when new problems come up
  2. Everyone who is connected to a problem needs to be at the session.
  3. At the beginning of each session, take a few minutes to explain what the process is for and how it works for benefit of those who are new to it. Use an example of a successful Five Whys session from the past.

Startup teams require three structural attributes

The structure is merely a pre-requisite, not a guarantee success

Scarce but secure resource

Too much budget or too little are both harmful for startup. Startup requires much less capital overall, but that capital must be absolutely secure from tampering.

Independent development authority

They have to be able to conceive and execute experiments without having to gain an excessive number of approvals. Handoffs and approvals slow down the Build-Measure-Learn feedback loop and inhibit both learning and accountability.

A personal stake in the outcome

This can be achieved through stock options or other forms of equity ownership. Use a bonus system, need not to be financial, as nonprofits organisation.
Eg, Shusa (主查) (Toyota’s term) – Vehicle under development of a shusa is referred as shusa’s car. Shusa has final, absolute authority over every aspect of vehicle development.

Innovation sandbox

  1. Any team can create a true split-test experiment that affects only the sandboxed parts of the product or service, or only certain customer segments on territories. However,
  2. One team must see the whole experiment through from end to end
  3. No experiment can run longer than a specified amount of time (usually a few weeks for simple feature experiments, longer for more disruptive innovations)
  4. No experiment can affect more than a specified number of customers (usually expressed as a percentage of the company’s total mainstream customer base)
  5. Every experiment has to be evaluated on the basis of a single standard report of five to ten (no more) actionable metrics
  6. Every team that works inside the sandbox and every product that is built must use the same metrics to evaluate success
  7. Any team that creates an experiment must monitor the metrics and customer reactions (support calls, social media reaction, forum threads, etc.) while the experiment is in progress, and abort it if something catastrophic happens

The innovation team should be cross-functional and have a clear team leader, like the Toyota shusa.

It should be empowered to build, market, and deploy products or features in the sandbox without prior approval. It should be required to report on the success or failure of those efforts by using standard actionable metrics and innovation accounting.

As an internal startup grows, entrepreneurs who created the original concept must tackle the challenge of scale.

  • New mainstream customers are acquired and new markets are conquered, the product becomes part of the public face of the company, with important implications for PR, marketing, sales, and business development.
  • The product will attract competitors: copycats, fast followers, and imitators of all stripes.
  • To combat the inevitable commoditization of the product in its market, operational excellence a greater role. This may require different type of manager who excels in optimization, delegation, control, and execution.
  • Investment in facilities, infrastructure, and loyal customers.

When changing the traditional work environments to Lean Startup, you will face some challenges.

Charging the definition of productivity for a team from functional excellence to validated learning will cause problems. Functional excellence can be excellence in marketing, safes, or product development. Functional specialists are accustomed to measure their efficiency by looking at the proportion of time they are busy doing their work. Eg, a programmer expects to be coding all day long. And Lean Startup will frustrate these experts, because there are constant interruption of meetings, cross-functional handoffs, etc.

Learn Startup is a framework, not a blueprint of steps to follow

It is designed to be adapted to the conditions of each specific company, rather than copy what others have done.

“In the past the man has been first; in the future the system must be first” (Taylor, 1911. The Principles of Scientific Management)