Rename files according to date


I recently wrote a Perl script, that renames the files in a directory according to the date, in the format “YYYYMMDD ##” where “##” is the running number.

Rationale

Because I used to download the photos using the mobile apps like Weibo or Twitter, however the file names are almost random. This made me hard to organize these photos on my computer.

The artists (or celebrities) usually share a set of their photos, so when I download these photos, the files should have mtime (modified time) in the correct order.

Yet, I don’t need to rename the file to the time precision like “HH:MM:SS”. I just need the date and followed by the running number, because it looks shorter.

Though we can just use the file browsers to sort the files according to the time, it is still inconvenient to browse the images by changing the sorting condition. Furthermore, mtime can be changed, and this will void the purpose of the sorting.

Lastly, the randomized filename is just meaningless to me. Rename them according to the date is much more useful, in my opinion.

 

Script

 

Why Perl?

In my opinion, Perl is less famous like Python in the present day. But I prefer to use Perl, due to the popularity in most Linux distribution. For example Perl is the base package of Arch Linux. Once I installed Arch Linux, I can run Perl script immediately.

Though Python is great, backward incompatibility sometimes causes issue, which I may need to maintain the script. If I write with Perl, I can pay less effort to maintain the script.

 

WARNING! And usage

As the script mentioned,

Rename is irreversible. Recommend to make a backup.

The usage is,

./rename_files_to_date.pl ./target_dir

Where target_dir is the directory that contains the files you want to rename. It will not rename the files recursively.

PLEASE USE THE SCRIPT AT YOUR OWN RISK!!!

After renaming the files, a log file will be created. It is used just in case you want to revert the file name. (But you have to do this manually.)

Advertisements

MySQL and PostgreSQL GUI tools


I am using Arch Linux, and recently there are some issues on MySQL Workbench and pgAdmin 4.

MySQL Workbench with the latest version 8.0.12 has issue to make connection through the SSH tunnelling. As a result, I have to downgrade to version 6.3.10. However, with the recent upgrade of icu package to 62.1, it breaks the MySQL Workbench 6.3.10. In order to solve this, I duplicate the following files,

cp libicudata.so.62 libicudata.so.61
cp libicui18n.so.62 libicui18n.so.61
cp libicuuc.so.62 libicuuc.so.61

So, it is a temporary solution until the package is fixed.

Then, for the PostgreSQL, I use pgAdmin. But again, with the recent upgrade of Python to version 3.7, which causes pgAdmin fails to start. I tried to run pgAdmin through Python 3.6 using virtualenv , but it doesn’t work. So, I have tried other solutions.

I tried Adminer. It depends on PHP and Apache server, similar to phpMyAdmin. However, it is less intuitive, compared to phpMyAdmin. Because Adminer doesn’t allow inline editing value.

So, I tried SQL Workbench/J. It requires Java and PostgreSQL JDBC. Once installed required package, it can work perfectly. Similar to Adminer, SQL Workbench/J can work with database other than PostgreSQL, such as MySQL.

 

Catechism and the Pope


I am neither theologian nor bible scholar. But just a Catholic layperson and would like to share my opinions.

The most recent hot issue is the changing of the Catechism about death penalty by Pope Francis. Then throughout the Twitter, I can see there are a lot of debates on this issue, including some people criticising the Pope.

Therefore, in this post, I will share some of my understandings.

Regarding the Catechism of the Catholic Church (CCC), according to USCCB Question 17,

Because the Catechism presents Catholic doctrine in a complete yet summary way, it naturally contains the infallible doctrinal definitions of the popes and ecumenical councils in the history of the Church.

However, there were interesting articles about authority of the CCC and non-infallible teachings, which should be read as well.

Next, Pope is not a saint. Throughout the Church history, there were some bad popes.

There are some Twitter users share about Summa Theologiae article whether it is lawful to kill sinners. However, we must bear in mind that Summa Theologiae is a great theological work, but it is not infallible.

Since I am not an expert, I cannot discuss more about the death penalty.

However, I would like to share some of the related chapters and verses from the Bible.

St John Gospel chapter 8 mentioned about an adulterous woman, which according to the Law of Moses, she should be stoned to death. However, Jesus responded to the crowd, “Let the one among you who is without sin be the first to throw a stone at her”.

Therefore, I am asking myself, if I support death penalty, what is my actual intention? Because of my hatred and my anger towards someone? Or is it because, if my beloved is murdered, I can revenge in the name of justice? And also, if I oppose death penalty, what is my actual intention? Do I just simply follow whatever Pope says?

In Gal 2:11, it mentions that Apostle Paul “opposed [Peter] to his face because he clearly was wrong”. Therefore, I don’t think it is wrong to oppose the Pope, if the Pope is wrong. However, we must be careful to ourselves, because the way we oppose may lead to more confusion, especially when we are using social network and we have some followers. And one should pray that he or she is not opposing the Pope because of pride.

In 1 Cor 1:12-13, it says that Corinthians were dividing themselves, some say they belong to Paul, some belong to Apollos, some belong to Cephas, and some belong to Christ. Similarly, this kind of behaviour continues until today. Therefore, as Catholics, we should continually pray for the Pope and the unity of the Church.

Firefox batch download extension: DownloadSelected


In my previous post, I mentioned about DownThemAll on Firefox 56. Now the latest release of Firefox is version 61, but there is no update of DownThemAll for Firefox Quantum support. Using older unsupported version like Firefox 56 is not a good idea, because there will have no security update. Firefox ESR 52 is slower than Firefox 56 as I have mentioned.

Since I have spare time recently, I spent some days to write a Firefox extension, DownloadSelected, to solve my problem.

DownloadSelected demo

The screenshot above shows a list of URLs, which I used Greasemonkey script to generate. Then highlight, right-click, and DownloadSelected. I don’t write any UI elements to indicate the download progress, but I put the progress to the console. Once the files are downloaded, they will be archived into a zip file and a Save As dialog will be shown. The source code can be found here.

This is not a replacement or alternative extension to DownThemAll, but this extension solve my fundamental problem. Main features are:

  1. Bulk download
  2. Download only selected text
  3. Downloaded filenames are based on HTML text instead of URLs
  4. Quantum and Google Chrome compatible

 

Inheritance and composition


The modern JavaScript with the ES6 syntax and the rise of the popularity like ReactJS, functional programming becomes more and more common. When using React, one of the common practice is to use composition instead of inheritance.

Because I started learning programming when the OOP was the most prevailing paradigm, I was trained to solve the problem by using OOP concepts like polymorphoism, inheritance, encapsulation, etc.

I think JS is the most interesting programming language in the modern technology. It supports server-side and client-side development. With the ES6, it supports OOP keywords like class and also using FP (functional programming) syntax like fat arrow (=>).

In OOP, the most common usage is the inheritance and polymorphoism. The following is an example of inheritance in JS,

class Shape {
  constructor(w, h) {
    this.width = w;
    this.height = h;
  }
  area() {
    return this.width * this.height;
  }
}

class Rectangle extends Shape {
  constructor(w, h) {
    super(w, h);
  }
}

class Triangle extends Shape {
  constructor(w, h) {
    super(w, h);
  }
  area() {
    return super.area() / 2;
  }
}

function main() {
  const rectangle = new Rectangle(4, 5);
  const triangle = new Triangle(4, 5);
  console.log('Rectangle area: ', rectangle.area());
  console.log('Triangle area: ', triangle.area());
}

main();

The shape area calculation can be re-written to composition instead of inheritance as followings,

class Rectangle {
  constructor(w, h) {
    this.width = w;
    this.height = h;
  }
  area() {
    return this.width * this.height;
  }
}

class Triangle {
  constructor(w, h) {
    this.width = w;
    this.height = h;
  }
  area() {
    const rect = new Rectangle(this.width, this.height);
    return rect.area() / 2;
  }
}

Therefore, Rectangle and Triangle do not inherit from Shape. In fact, Triangle uses Rectangle to calculate the area. This is the object composition, and it is same as the way of composition in React. Furthermore, one of the greatest features of JS is closure. This allows React to pass a function with specific logic as a parameter to a generic component. Thus, the generic component can be designed without the prior knowledge of the business/application logic. This will produce a result similar to method override in OOP.

Moreover, the object composition can be re-written to function composition as FP.

const rectangleArea = (w, h) => w * h; // In math, f(x,y) = x * y
const halving = (area) => area / 2; // In math, g(x) = x / 2
const triangleArea = (w, h) => halving(rectangleArea(w, h)); // In math, h(x,y) = g(f(x,y)) = f(x,y) / 2

function main() {
  console.log('Rectangle area: ', rectangleArea(4, 5));
  console.log('Triangle area: ', triangleArea(4, 5));
}

main();

Scientific Learning


In my previous post, I wrote about Lean Startup. And I found that it is very interesting that I have come through these things.

  • S.M.A.R.T (specific, measurable, actionable, realistic, time-bound) (post)
  • Machine Learning – Deep Learning – Convolutional Neural Network (CNN)
  • Statistics
  • Kaizen (改善法) (post)
  • Agile development. Test development development (TDD)
  • Research study

When I read The Lean Startup (Ries, 2011), I learn about lean manufacturing. Then only I discovered that kaizen is also part of the lean manufacturing. Wow! I realise that, all of these things are related. And the core of the concept is, “scientific learning”.

In Lean Startup, in my opinion, the fundamental concept is Build-Measure-Learn loop. According to Ries (2011), the “Entrepreneurship is a management, learning, and involves experiments”. Throughout the whole entrepreneurship, it is about the learning. It is a scientific learning, because entrepreneurs need to form hypotheses and then test the hypotheses with experiments. Furthermore, the method can be replicated,  The results can be measured and validated. It is a validated learning. The main purpose of Lean Startup is to make the Build-Measure-Learn loop faster, then for each iteration, the product will be optimized and improved through the learning.

This is exactly how Machine Learning works. In Machine Learning, the core concept is the gradient descent. For each calculation of output (≈ build), the machine get the errors (difference of actual output and the desired output) (≈ measure). Then using the errors, machine will update the weight so that it will produce the actual output that is closer to the desired output (≈ learn). It involves small batches and many iterations throughout the training.

Agile development shares many characteristics with Lean Startup. It involves iterative development and daily stand-up meetings (scrum) which improves the communication. Besides that, TDD is also part of the agile development. Why is TDD important? I have wrote about it previously. But after reading The Lean Startup, I found that TDD is more than just making the product stable. As we use TDD, we can test the features or functions with command-line. It is extremely faster than testing the functionality by running the application from beginning to the end manually. Imagine that you want to test a webapp functionality, you have to open the web browser, login, click-and-click-and-click, refresh-refresh-and-refresh. If using TDD, as lean manufacturing, this can minimize the waste drastically. Similarly, you write the code (≈ build), you test it (≈ measure), and you make changes to optimize your code (≈ learn).

Unluckily, when I was doing PhD research study, I didn’t know about Build-Measure-Learn. Research study also shares a characteristic of entrepreneurship, i.e. uncertainty. However, getting supervisors’ feedback is not an easy job, as most of the time our supervisors are hardly to give instant response. As a result, unlike entrepreneurship, research study has difficulty to accelerate the Build-Measure-Learn loop. And most of the time, research study requires enormous prior knowledge of the research area.  Without fulfilling this requirement, supervisors’ feedback doesn’t help much at all.

When I was a lecturer, my institution adopts OBE (Outcome-Based Education). It shares the characteristics as Lean Startup and S.M.A.R.T, as the outcomes have to be measurable and observable. However, I personally don’t agree the ways of implementation of OBE, because it makes the teaching become very rigid, especially setting up the exam papers have to fulfil the criteria strictly. I agree that the outcome should be measurable, yet the ways of implementation is exhaustive and not really effective. It is muda and muri. But I believe that, if students are able to implement Build-Measure-Learn loop in learning, the learning will become effective. For example, “build” the knowledge through lecture class, group discussion, and other activities. “Measure” their gained knowledge through various tests. “Learn” from the results of the tests. And, repeat. Each iteration should be small. By each iteration, the students should able to learn through the failures they made and optimize their solution to get the desired result. (Nevertheless, I believe that this implementation is impractical, due to the cultural pressure and stereotype that the students results should be a bell-curve. W*F!)

As a conclusion, by using the scientific learning method like Lean Startup, kaizen, or even Machine Learning in our daily life (small batches, iterative, and measurable), surely this will make our life better.

After reading The Lean Startup


Because I was planning to start a small project, I have developed a prototype. And I asked a friend for some advice and showed the project to him. Then he gave me his opinions, and asked me to read The Lean Startup (Eric Ries, 2011). So, I stopped my project, and start reading the book. This post is about what I get from the book.

(I will write another post for what I have learnt so far.)

Lean Startup

Entrepreneurship is a management, learning, and involves experiments.

Challenges:

  1. Build an organization that can test the assumptions systematically.
  2. Perform the rigorous testing without losing sight of the company’s overall vision.

Methods:

  1. Entrepreneurs everywhere
  2. Entrepreneurship is management
  3. Validated learning
  4. Build-measure-learn
  5. Innovation accounting. Accounting here refers to accountability.

Vision -> Strategy -> Product

Startup has an engine, namely engine of growth. So, the entrepreneurship is to make the business grow.
Because it is a build-measure-learn model, just like driving to the destination, we need to know whether to pivot for a sharp turn or persevere with the current path.
So, we can optimize our product at the end.

Hypotheses

Value hypothesis and Growth hypothesis

“Unfortunately, because customers don’t really know what they want, it’s easy for these entrepreneurs to delude themselves that they are on the right path.”

Minimum viable product (MVP)

Should communicate with the targeted first adopters, then only can define our MVP.

Actionable metrics vs vanity metrics

Genchi gembutsu – “go and see for yourself”

Customer archetype

A brief document that seeks to humanize the proposed target customer. It is an essential guide for product development and ensures that the daily prioritization decisions aligned with the customer to whom the company aims to appeal.

Lean UX

Need not to worry the ideas will be stolen by other companies.

  1. The managers in most companies are already overwhelmed with good ideas. Their challenge lies in prioritization and execution, and it is those challenges that give a startup hope of surviving.
  2. If competitor can outexecute a startup once the idea is known, the startup is doomed anyway.
  3. The important thing is that entrepreneur can accelerate through the Build-Measure-Learn feedback loop faster than anyone else can. If this is true, it makes no difference what the competition knows. If it is not, the startup has much bigger problems. Secrecy doesn’t fix the problems. It will be competed by fast followers.

Launch MVP under different brand name, to avoid damaging the company’s established brand

Startup is

  1. rigorously measure where it is right now, confronting the hard truths that assessment reveals
  2. devise experiments to learn how to move the real numbers closer to the ideal reflected in the business plan

Innovation accounting – Three learning milestones

  1. Use MVP to establish real data to know where the company right now is, so that you can track your progress
  2. Tune the engine from the baseline toward the ideal, using micro changes and product optimizations
  3. Pivot or persevere
    a. If pivot, needs to reestablish a new baseline and then tuning the engine
    b. Successful pivot should show that the engine-tuning activities are more productive than before

Prototype

Smoke test

Measure one thing: whether customers are interested in trying a product. It is insufficient to validate an entire growth model. But it is useful to get feedback on this assumption before committing more money and other resources to the product.

When choosing among the many assumptions in a business plan, it makes sense to test the riskiest assumptions first.

Good design is one that changes customer behaviour for the better. Else, the design should be judged a failure.

Cohort analysis.

Leap-of-faith metrics.

Extreme Programming, agile development

  • Sprint (n-period iteration cycles)
  • Prioritize the work to be done by writing a series of user stories (instead of technical specification)

Split-testing, aka A/B testing

Get feedbacks through interview and survey.

Kanban

stories are assigned to 4 categories: backlog, in progress, done, validating

3 A’s of metrics: actionable, accessible, auditable

  • Actionable – antidote to vanity metrics. Vanity metrics doesn’t demonstrate clear cause and effect.
  • Accessible – antidote to misuse of data, ie too many reports that cannot be understood by the employees and managers. Cohort-based reports turn the complex actions into people-based reports. “Metrics are people too”. Analytics or the data should be part of the product itself.
  • Auditable – antidote to blaming when failure. “Metrics are people, too”. Talking to customers to check if the reports contain true facts. By this, can gain insights into why customers are behaving the way the data indicate. The reports should be drawn directly from the master data, instead of intermediate system, to reduce opportunities for error.

Pivot or persevere

  • The more money, time, and creative energy that has been sunk into an idea, the harder it is to pivot.
  • Failure is a prerequisite to learning.
  • zoom-in-pivot – refocusing the product on what previously had been considered just one feature of a larger whole.
  • zoom-out-pivot – reverse of zoom-in-pivot
  • customer segment pivot – keeping the functionality of the product the same, but changing the audience focus.
  • customer need pivot
  • platform pivot
  • business architecture pivot, value capture pivot, engine of growth pivot, channel pivot, technology pivot
  • A pivot is better understood as a new strategic hypothesis that will require a new MVP to test.
  • The startup has to find ways to achieve the same amount of validated learning at lower cost or in a shorter time.

Reasons inhibit pivots

  • Vanity metrics – false conclusions
  • Unclear hypothesis – impossible to experience complete failure, without failure, there is no impetus (momentum) to perform a pivot
  • Afraid – acknowledging failure leads to low morale

Meeting. Frequency depends. Should involve both product development and business leadership teams.

  • Product development team must bring a complete report of the results of its product optimization efforts over time as well as a comparison of how those results stack up against exceptions.
  • Business leadership team should bring detailed accounts of their conversations with current and potential customers.

MVP is for the early adopters. Next level is the mainstream customers. They have different requirements and are much more demanding. They are less forgiving.

Single-piece flow (lean manufacturing)

It seems inefficient, but it has been confirmed in many studies that it is faster way to get a job done.

“Toyota Production System (TPS) is probably the most advanced system of management in the world, but even more impressive is the fact that Toyota has built the most advanced learning organization in the history.”

Sustainable growth – New customers come from the actions of the past customers.

  • Word of mouth
  • As a side effect of product usage
  • Through funded advertising
  • Through repeat purchase or use

3 kinds of engine of growth: Sticky, viral, and paid

Though a business can have multiple engines of growth at a time, it is recommended to focus on just one engine of growth.

Training programme

Standardise our work processes and prepare a curriculum of the concepts that new employees should learn.

Andon cord

“Stop production so that production never has to stop”. Brings work to a stop as soon as an unccorrectable quality problem surfaces, which forces it to be investigated. You cannot trade quality for time, else the resulting defects will slow you down later. Defects cause a lot of rework, low morale, and customer complaints

The logic of validated learning and the MVP says that we should get a product into customers’ hands ASAP and that any extra work we do beyond what is required to learn from customers is waste. However, the Build-Measure-Learn feedback loop is a continuous process. We don’t stop after one MVP but use what we have learned to get to work immediately on the next iteration.

When we go too fast, we will cause more problems. Adaptive processes force us to slow down and invest in preventing the kinds of problems that are currently wasting time.

Five Whys

Build an adaptive organization – Asking “Why?” five times to understand the root cause. This help to uncover the root problem and correct it. Using this technique is possible to reveal that a purely technical fault is actually caused by human managerial issue.

  • However, the investment (fix) should be smaller when the symptom is minor, and larger when the symptom is more painful. We don’t make large investments in prevention unless we are coping with large problems.
  • Startup teams should go through the Five Whys whenever they encounter any kind of failure, including technical faults, failures to achieve business results, or unexpected changes in customer behaviour.
  • Proportional investment

Avoid to making Five Whys into Five Blames which pointing fingers at each other and trying to decide who is at fault. To avoid this,

  • make sure that everyone affected by the problem is in the room during the analysis of the root cause
  • The meeting should include anyone who discovered or diagnosed the problem, including customer service representatives who fielded the calls, if possible.
  • should include anyone who tried to fix the symptom as well as anyone who worked on the subsystems or features involved.
  • If the problem was escalated to senior management, the decision makers who were involved in the escalation should be present as well.

If blame is inevitable, the most senior people in the room should accept this: if a mistake happens, shame on us for making it so easy to make that mistake.

Five Whys analysis is to have a systems-level view as much as possible.

(Interesting point. If a new employee can easily break the system, it is not the new employee to be blamed, but the whole team fault.)

Five Whys requirements:

  1. Be tolerant of all mistakes the first time.
  2. Never allow the same mistake to be made twice.

Don’t use Five Whys to solve “baggage” issues

  1. Use it when new problems come up
  2. Everyone who is connected to a problem needs to be at the session.
  3. At the beginning of each session, take a few minutes to explain what the process is for and how it works for benefit of those who are new to it. Use an example of a successful Five Whys session from the past.

Startup teams require three structural attributes

The structure is merely a pre-requisite, not a guarantee success

Scarce but secure resource

Too much budget or too little are both harmful for startup. Startup requires much less capital overall, but that capital must be absolutely secure from tampering.

Independent development authority

They have to be able to conceive and execute experiments without having to gain an excessive number of approvals. Handoffs and approvals slow down the Build-Measure-Learn feedback loop and inhibit both learning and accountability.

A personal stake in the outcome

This can be achieved through stock options or other forms of equity ownership. Use a bonus system, need not to be financial, as nonprofits organisation.
Eg, Shusa (主查) (Toyota’s term) – Vehicle under development of a shusa is referred as shusa’s car. Shusa has final, absolute authority over every aspect of vehicle development.

Innovation sandbox

  1. Any team can create a true split-test experiment that affects only the sandboxed parts of the product or service, or only certain customer segments on territories. However,
  2. One team must see the whole experiment through from end to end
  3. No experiment can run longer than a specified amount of time (usually a few weeks for simple feature experiments, longer for more disruptive innovations)
  4. No experiment can affect more than a specified number of customers (usually expressed as a percentage of the company’s total mainstream customer base)
  5. Every experiment has to be evaluated on the basis of a single standard report of five to ten (no more) actionable metrics
  6. Every team that works inside the sandbox and every product that is built must use the same metrics to evaluate success
  7. Any team that creates an experiment must monitor the metrics and customer reactions (support calls, social media reaction, forum threads, etc.) while the experiment is in progress, and abort it if something catastrophic happens

The innovation team should be cross-functional and have a clear team leader, like the Toyota shusa.

It should be empowered to build, market, and deploy products or features in the sandbox without prior approval. It should be required to report on the success or failure of those efforts by using standard actionable metrics and innovation accounting.

As an internal startup grows, entrepreneurs who created the original concept must tackle the challenge of scale.

  • New mainstream customers are acquired and new markets are conquered, the product becomes part of the public face of the company, with important implications for PR, marketing, sales, and business development.
  • The product will attract competitors: copycats, fast followers, and imitators of all stripes.
  • To combat the inevitable commoditization of the product in its market, operational excellence a greater role. This may require different type of manager who excels in optimization, delegation, control, and execution.
  • Investment in facilities, infrastructure, and loyal customers.

When changing the traditional work environments to Lean Startup, you will face some challenges.

Charging the definition of productivity for a team from functional excellence to validated learning will cause problems. Functional excellence can be excellence in marketing, safes, or product development. Functional specialists are accustomed to measure their efficiency by looking at the proportion of time they are busy doing their work. Eg, a programmer expects to be coding all day long. And Lean Startup will frustrate these experts, because there are constant interruption of meetings, cross-functional handoffs, etc.

Learn Startup is a framework, not a blueprint of steps to follow

It is designed to be adapted to the conditions of each specific company, rather than copy what others have done.

“In the past the man has been first; in the future the system must be first” (Taylor, 1911. The Principles of Scientific Management)