RSS Feed

ROC, AUC, WTF?

Posted on

These few days I was spending my whole time to understand this ROC (receiver operating characteristic) curve. In machine learning, ROC is a very common way to evaluate the prediction performance. The AUC (area under curve) of ROC indicates the accuracy of prediction of a classifier.

If you wish to learn more, these two links are the best resources: here and here.

I searched through tutorial and Q and A sites on how to do the plot of ROC and calculating the AUC. The answers were telling me about “cut-off”, “threshold”, or some weird terms. And some answers were telling me to use R package to plot the graph. WTF? I know all of these things. My question was, “How can I plot ROC curve with my classifier?!”

So, there was a real gap between what I had known and the problem I faced. The gap was the classifier that I created cannot be used to plot the ROC curve directly. Because my classifier is a discrete classifier. That is, it just predicts with a label, then checks with actual label whether the prediction is true or false. Though, I can improve this result to true positive, false positive, true negative, and false negative, it can only produce one point in the ROC space.

In order to plot the curve, I need a theshold that can be manipulated to produce a sequence of TPRs (true positive rates) and FPRs (false positive rates). If I have a discrete classifier, I can never produce the sequence of TPRs and FPRs. That is my real problem.

In order to solve this, the only solution is to transform my classifier to ranking or scoring classifier. This is the crucial part that solves my problem. Now, the question is, what is the score here? I believe all classifiers involve some calculations to get some values, then only do the prediction whether the input should be labelled as true or false, even it is a discrete classifier. So, that calculated value is the score! Even if your classifier does a random guess, the random value can be used as the score.

As a result, I need to discard the discrete classification when plotting ROC curve. But the discrete classification is stilled being used during the training. So, by adjusting the theshold, for any input that produces the score greater than the theshold will be labelled as the positive. And from here, I just need to check whether it is true positive or false positive. Continuously adjusting the threshold, I can get a sequence of TPRs and FPRs. Yay. No more discrete classification, and the ROC curve produced.

P/S: Maybe this concept is just as simple as ABC, that is why nobody cares and mention it online.

AAC file re-visit


In my previous post, I mentioned about AAC and the ID3 tag. And I mentoined that

I have an AAC audio file (technically M4A) […]

I used Audacious previously, then change to DeadBeef. The main reason I changed was because I kept failing to play AAC audio file. What’s wrong? FFplay can play it, SMPlayer can play it, DeadBeef can play it, Clementine can play it, but Audacious cannot. Audacious has a AAC plugin, it should support AAC format.

But if I play the AAC file with Audacious, I will get this error,

Unknown playback error (check the console for detailed error information)
ERROR aac.cc:373 [play]: No valid frame header found.

Seach online and found that, it is the file extension issue. So, what is it?

Using the “exiftool”, I found that the MIME Type is video/mp4. That is the fix!!

I renamed the AAC file to Mp4. Now Audacious can play the file well.

Now, when it was in the AAC extension, in order to view the audio metadata such as author, title, ablum, I used Kid3 to edit the APE instead of ID3v2, so that FFplay can play it. ID3v2 is actually for the MP3 format. If use ID3v2 on the AAC extension audio file, FFmpeg cannot convert it and FFplay and SMPlayer both failed to play the audio file properly.

But after I rename the file into MP4, there is an issue. MP4 metadata (I think it is also ID3v2) cannot be written using Kid3, due to the existence of APE tag. (Kid3 doesn’t mention anything about this, I solve this heuristically.)

  1. So, I need to remove the APE tag using Kid3 by naming the file as AAC first.
  2. Then, I rename it to MP4 and use Kid3 to edit the ID3v2

As a conclusion, I use back Audacious as my primary audio player. It is the best!

(Why not DeadBeef? DeadBeef lack of the feature to copy-paste the songs from one playlist to another.)

AAC audio file and ID3 tag


I just found that, if I have an AAC audio file (technically M4A), and if I added the ID3 tag 2 (aka ID3v2), then the audio file will failed to be converted by ffmpeg.

It can be either converted to mp3

  • using DeadBeef audio player, or
  • remove the ID3v2 tag then convert

So, how to add the metadata like ID3 tag? Use the Kid3 and add the Tag 3 (aka APE tag). This will not affect how ffmpeg to read the file.

Dell Vostro 5459 hibernation


In the previous post (1 year ago), I mentioned the hibernation issue. I believed that it was related to the NVidia graphic card. Related forum can be found here.

But these few days, I notice that whenever I shut down the laptop, it will show the systemd messages. Previously, if I did suspend my laptop, then resume, then shut down will show only black blank screen, until the power off. I believe that the graphic card issue is being fixed with the recent update.

I am now using linux-lts 4.9.13-1 and nvidia-dkms 378.13-2

A brief comparison of GTK+ and Qt


I used to like C language, because it is a basic of programming, and it is portable, and it is low-level. When writing program with C language, it is just like showing off your advanced programming skill, how you manage the memory, how you manage the pointers and creating the linked list. However, in terms of efficiency, C++ is much more powerful, because of object-oriented and the syntax.

Because I like C language, so I chose GTK+ over Qt for long time ago. Not only that, I am also fond of GTK+ desktop environments like GNOME, Xfce4, LXDE, Cinnamon, but not Mate. I feel that KDE is heavy weight.

One of my personal projects, Med, which was written with GTK+, had some issues with multi-threading. I believe that my engine part does not have any problem. As a result, the suspicious part is the GTK+. The program crash without symptoms, and difficult to reproduce. Therefore, I decided to re-write the UI with Qt. If, Qt also produces the same problem, meaning my algorithm is problematic.

In my opinion, GTK+ is straight forward, because of procedural paradigm. Therefore, it is easy to learn and implement. The UI can be design with Glade. But I feel that it is still lack of something, such as button click callback function. Besides that, though there is gtkmm (C++ interface of GTK+), the library like WebKitGTK+ does not have the documentation for gtkmm (I believe still can work). In summary, GTK+ is slower as writing the code in C language.

On the other hand, Qt is more complicated. To use it, better to use qmake project file to generate the Makefile, or use the CMake, so that we can include the necessary headers and link with necessary libraries. And we also need to use some macro. This makes me feel that the coding is very Qt-oriented, not just a C++ language. However, in terms of design, I feel that Qt Designer is much easier. But GTK+ and Qt layout uses different concept.

Though GTK+ and Qt are both object oriented, GTK+ is in the library level yet Qt is in the language level. It is easier to write inherited classes in Qt, and easily make changes at the language level.

As a conclusion, Qt is better than GTK+ for development as C++ is better than C.

Heading, anchor, and bookmarking


Sometimes I read online articles, and these articles are usually long pages and have outlines at the beginning. These outlines are the hyperlinks to the subtopics headings. Technically, you click the outline hyperlink, your browser will browse to the “anchor”, the URL will append with hash (#). Therefore, it is useful for bookmarking, so that you can share the URL target on the topic to someone else, or re-visit your bookmark.

Thus, I wrote a script as bookmarklet to solve this issue, so that I can click the headings when I am reading the article. You can create the bookmark with the following URL:

javascript:var elems=document.querySelectorAll("h1[id],h2[id],h3[id],h4[id],h5[id],h6[id],h1>*[id],h2>*[id],h3>*[id],h4>*[id],h5>*[id],h6>*[id],a[name]:not([href])");var array=Array.from(elems);array.forEach(function(elem){elem.style.cursor="pointer";elem.setAttribute("title","anchor:"+(elem.id?elem.id:elem.name));elem.innerHTML+=" *";elem.addEventListener("click",function(el){location.hash=el.target.id?el.target.id:el.target.name})});

(Yes, it starts with javascript: instead of http://. Sadly WordPress.com doesn’t allow to create bookmarklet in the post. Moreover, I was actually using ⚓ instead of asterisk *, but WordPress.com auto convert it to emoji, consequently cannot use it in the script above.)

You can try any Wikipedia page to test it. It also works on Stackoverflow, so that you can share the URL to a certain answer to others.

Lecturer, researcher, hobbyist, and software developer


I am cognitive science student. That is why I learnt AI, computational linguistic, machine learning, expert system, etc. 

Since I was a researcher on Augmented Reality, then I applied my computing skills. After this, I became a non-computer related lecturer, and spent my time doing programming as a hobbyist. Then later I became computer science lecturer, yet still had to do programming as a hobbyist. Now as a software developer, I know what are the differences of these roles: lecturer, researcher, hobbyist, and a true software developer. 

A lecturer, as long as they know how to bluff, they are considered good in programming skill.

A researcher, whether they are good in programming skill is totally not important. Because the focus is the research. As a researcher, you do not need to write programs  or scripts. Though these may help, they are not required. As long as your research methodology is correct, that is the core element of the research.

As a hobbyist, you can learn any programming language and write any program. You can write the code that you yourself think it is elegant. You can write any powerful function and use a simple but accurate solution to solve your own problem. You can share your code and publish your project. You can have very strong knowledge to write wondrous algorithm and solve the problem effectively and efficiently. BUT, you are still lacked of industrial experience. 

An actual software developer who works in industrial, what you need is to produce stable products. Yes. Stable. In order to produce stable products, you need to have your products fully tested. Hire a tester can only perform black box testing. They test only as an end user perspective. But the problem is, the bugs are found after you have written the dependent codes on the bugs. So, it would be best to test before these, during the development level. As a developer, writing test cases becomes exhaustive. That is why, Test Driven Development comes in. A stable product is the actual thing that the customers want. They paid the money to get the product, or specifically, the customers hire you to develop a product to them. That is why your products must be stable. Unlike the commercial products, the government projects usually have no clear objective. As long as you deliver a prototype, you can get the money. That is why, the products standards are very different, the developers quality is also very different.

As a lecturer, you talk.

As a researcher, you do experiments.

As a hobbyist, you learn and write.

As an actual software developer, you write and test.