Importing Windows Live Calendar to Google Calendar

Windows Live is a great calendar, but it is good to know you have the option to migrate to Google Calender, should you desire to do so. The process is quite simple:

  1. Navigate to your Windows Live Calendar account
  2. Click Share and select the calendar you want to import
  3. In the left pane, click get a link 
  4. Under Show event details click Create
  5. Copy the link under Import into other calendar applications (ICS)
  6. Change webcals:// to http:// and paste the link in your browser to download it
  7. Save the ICS file to some location on your computer as calendar.ics
  8. If you have non-ASCII characters in your events (in my case I have Hebrew characters) you must mark the ICS file as UTF:
    1. Open calendar.ics in Notepad++
    2. In the top menu click Encoding -> Encode in UTF-8-BOM
    3. Save the file and exit Notepad++
  9. Navigate to your Google Calendar account
  10. Click the dropdown next to My calendars and select Create new calendar
  11. Fill out the calendar fields for the new calendar and click Create Calendar
  12. Click the dropdown next to Other calendars and select Import Calendar
  13. Click Choose File and select the ICS file you downloaded above (calendar.ics)
  14. In the Calendar dropdown select the name of the calendar you just created
  15. Click Import

You’re all done!
Note however that event reminders are not imported 🙁

HowLongToBeatSteam

Ever wonder how long it would take you to complete your entire Steam library? I know I did about 10 months ago. Alas, the only site that would answer me at that time, Nate Collings‘ cool SteamPlaytime, has closed up shop.

CHALLENGE ACCEPTED.

After many weekends and vacations of hard work, I present to you HowLongToBeatSteam.com:

hltbs

Features

  • Examine your total current and remaining playtimes in the different HowLongToBeat completion levels (Main, Extras, Completionist).
  • Slice and filter playtime by name, genre and Metacritic score.
  • Include and exclude specific games in the calculation.
  • Examine your full playtime table and sort by current playtime and time to beat.
  • You can even check out an imaginary profile containing all of Steam’s games (currently over 9,000!)

Enjoy 🙂

Client stuck on “Awaiting Challenge” trying to connect to ioquake3 server

Every week, I meet up with a group of friends from work for a LAN party. Since the age-old Quake vs Unreal debate could not be settled (obviously Quake is better and they are wrong), we decided to alternate between Quake3 and Unreal 2004.

Last week was Quake’s turn, so I naturally set up a dedicated server. Alas, even though most people were able to connect and play without issue, some got stuck in the dreaded “Awaiting Challenge” stage and could not connect. The Unreal fans rejoiced, believing this spelled the end of our quake sessions.

But my resolve when it comes to Quake should not be underestimated…

After a long process of elimination and a chat conversation with one of the ioquake3 developers (DevHC – great guy), the issue was identified. It turns out that when specifying the host name (rather than the IP) of the server, the ioquake3 client resolves it to IPv6 (where available) by default. For some reason, this didn’t resonate well with the server. Once we specified the IPv4 explicitly, the issue was resolved.

You have several options to work around this issue:

  1. Specify the IPv4 address of the server explicitly
  2. Force IPv4 resolution by issuing the console (~) command /connect -4 hostName
  3. Disable IPv6 in the client by adding the following lines to your autoexec.cfg:
    set net_enabled 1
    net_restart

Happy fragging!

In C++, even a thread is not thread-safe (or: why you should use tasks part 2)

Consider the following (contrived) code:

using namespace std;

void work()
{
    this_thread::sleep_for(chrono::seconds(2));
}

void wait(thread* t)
{
    t->join();
}

int main()
{
    thread worker(work); 
    thread waiter1(wait, &worker); 
    thread waiter2(wait, &worker);

    waiter1.join();
    waiter2.join();

    return 0;
}

The intent of the code is quite clear – we spawn a worker thread and have two threads wait for it to finish. Let’s assume that the sleep is long enough so that both threads try and join while the worker is still running (otherwise the thread stops being joinable and calling join throws). Even under that assumption, the code’s behavior is not defined (UB). The reason is in the docs:

Note that any operations on the thread object itself are not synchronized (unlike the operations within the thread it represents).

In other words, the std::thread object itself is not thread safe, so we can’t call its methods concurrently (which is what we’re doing in waiter1 and waiter2). We’ll have to do something like this:

using namespace std;

mutex m;

void work()
{
     this_thread::sleep_for(chrono::seconds(2));
}

void wait(thread* t)
{
    lock_guard<std::mutex> lock(m);
    if (t->joinable())
    {
        t->join();
    }
}

int _tmain()
{
    thread worker(work);
    thread waiter1(wait, &worker);
    thread waiter2(wait, &worker);

    waiter1.join();
    waiter2.join();

    return 0;
}

Of course, were you using tasks, you wouldn’t have needed to concern yourself with such trivialities:

using namespace std;
using namespace concurrency;

auto worker = create_task([]
{
    this_thread::sleep_for(chrono::seconds(2));
});

auto waiter1 = create_task([&worker]
{
    worker.wait();
});

auto waiter2 = create_task([&worker]
{
     worker.wait();
});

waiter1.wait(); //works
waiter2.wait(); //great

I hope this has taken you one step closer to ditching bare threads (if you haven’t already). If it didn’t, be sure to check out (Not) using std::thread by Andrzej Krzemieński (his blog is great all around, so I recommend you check it out anyway).

Reminder:

The case of the crashing std::thread destructor and why you should use tasks instead of threads

I recently encountered an interesting crash in our iOS application at work. Here’s the relevant stack trace section:

…
libc++abi.dylib std::terminate()
libc++.1.dylib std::__1::thread::~thread()
…

Luckily, the documentation for std::~thread provides us with the root cause:
If *this has an associated thread (joinable() == true), std::terminate() is called.

Specifically, the joinable() documentation states:
A thread that has finished executing code, but has not yet been joined is still considered an active thread of execution and is therefore joinable.

Seeing as the thread in question was indeed never joined (or detached) we were hit by this error. Now, this issue can be solved by joining with or detaching from the thread, but I think that would be the wrong lesson to learn here.

This is just one example of how things are more difficult with threads than they are with tasks. Practically any imaginable scenario is made easier and less error-prone with tasks. Just as tasks were a paradigm shift in .NET, they should be in C++11 – only manipulate threads directly if you must!

Posting forms with Knockout to dynamic Sammy routes

Single-page applications (SPAs) are the standard these days, and two aspects are typically considered for their development:

  1. Data binding the view (HTML) with the model (Javascript)
  2. Navigating between logical “pages” without reloading.

There are many libraries that handle both needs, and I went with Knockout for (1) and Sammy for (2) – mostly because I was already familiar with them (they are both excellent).

The way Sammy works is by defining “routes” (typically hash tag routes for SPAs), which are basically URLs that trigger JS methods. For example, you might define the following route:

get('#/:name', function() {
    alert(this.params['name']);
});

When the URL changes to mysite.com/#/Ohad, an alert will pop up saying “Ohad”. Now, suppose we have a form we want to “submit” to the route above Sammy route:

<form
data-bind="submit:function(){window.location='#/'+name();}">
 <input type="text" data-bind="value: name"></input>
 <button type="submit">Submit</button>
</form>

The input’s data-bind attribute binds the input’s value to our view model’s name observable. The form’s data-bind attribute binds the form submission event to a URL redirection for the desired Sammy route. For more information about Knockout bindings, visit http://learn.knockoutjs.com.

The code above works, but not exactly as expected. See, when no action attribute is specified on the form element, the browser assumes the current URL without hashtags is the submission URL, and so right after the URL is redirected to mysite.com/#/MyName it is reverted to just mysite.com by the browser.

This messed up my hash tag navigation model, and nothing I tried prevented the browser from reverting the address (canceling the form submission event, returning false from the handler method, etc). And then it hit me – why fight the browser?

<form
data-bind="attr: { action: '#/'+name() }">
 <input type="text" data-bind="value: name"></input>
 <button type="submit">Submit</button>
</form>

See what we did there? Instead of using knockout to circumvent the normal form submission flow, I go with it. Whereas Knockout’s submit binding tries to cancel the browser’s submission event and replace it with the supplied method, here the submission flow is completely standard. We just need to make sure the action attribute points to the right place, and voila – hash tag history is preserved.

BTW, you may wonder why I insisted to go with a form, where in truth no “real” submission takes place here. Indeed, a regular div with a regular button would not have had this issue to begin with!

Well, the answer is a bit silly. I could not find any reliable cross-browser method to intercept the [enter] key on the input in order to trigger submission (everything is either deprecated or not globally supported). And frankly, even if I did, it would be a hack – the browser should handle such logic. Plus, I enjoyed the challenge 🙂

Static definitions in header files may cause malloc errors on application teardown

I recently encountered the following error in the shutdown procedure of XCTest (we’re using Kiwi which is based on it):

Malloc Error: pointer being freed was not allocated

Here’s how the stack looked like (truncated to section of interest):

(lldb) bt

frame #4: 0x00e12ae7 OurProduct`std::__1::basic_string<char16_t, std::__1::char_traits<char16_t>, std::__1::allocator<char16_t> >::~basic_string(this=0x042a2058) + 103 at string:2093

frame #5: 0x00e10cb7 OurProductI`std::__1::basic_string<char16_t, std::__1::char_traits<char16_t>, std::__1::allocator<char16_t> >::~basic_string(this=0x042a2058) + 23 at string:2090

frame #6: 0x0c94a7ac libsystem_sim_c.dylib`__cxa_finalize_ranges + 315

frame #7: 0x0c94a832 libsystem_sim_c.dylib`__cxa_finalize + 59

frame #8: 0x0c94ab55 libsystem_sim_c.dylib`exit + 57

frame #9: 0x20117684 XCTest`+[XCTestProbe exitTests:] + 57

As you can see, XCTest was executing its shutdown procedure, and during this finalization phase the destructors of static instances were executed. It was somewhat vexing that a static destructor would try to free up a pointer that was never allocated – after all, statics are allocated only once when the process starts up. And if some static was not defined (or defined multiple times), a linker error should have been thrown. How could this situation come to be?

Failing to answer that question armed with the data above alone, I proceeded to attempt and locate the faulty string in our code. I had hoped I could spot something unusual about it, that would hopefully hint me at the root cause of the failure.

(lldb) frame select 4
(lldb) frame variable *this

(std::__1::basic_string<char16_t, std::__1::char_traits<char16_t>, std::__1::allocator<char16_t> >) *this = {

__r_ = {

std::__1::__libcpp_compressed_pair_imp<std::__1::basic_string<char16_t, std::__1::char_traits<char16_t>, std::__1::allocator<char16_t> >::__rep, std::__1::allocator<char16_t> > = {

__first_ = {

= {

__l = (__cap_ = 17, __size_ = 14, __data_ = u”Helvetica Neue“)
__s = {

The string looked innocent enough. However, upon finding it in our code (thankfully the literal appeared as is, and only once), I noticed it was defined (initialized) in the header. Statics should be declared in the header – never defined. Code such as  Bar::foo = “Helvetica Neue” belongs in the cpp file. For more information see http://stackoverflow.com/questions/185844/initializing-private-static-members.

And indeed, moving the definition to the cpp resolved the issue. Now, the keen reader will notice that a linker error should have been thrown, rather than the runtime exception I encountered. I’m not exactly clear on why the former was not the case, but the prevailing theory entails a linker optimization that allocated only once for multiple compilation units, which backfired when the shutdown sequence tried to free all occurrences separately (or one occurrence that happened to not be the one that was initialized).

If you have a better explanation, I’ll be happy to hear it.
In the meantime, mind your statics 🙂

Debugging unobserved concurrency runtime task exceptions

The C++ Concurrency runtime (AKA PPLX) is the unmanaged answer to the Task Parallel Library (TPL), and it works surprisingly well. It is even cross platform by way of the C++ Rest SDK (codename Casablanca).

I work at Microsoft and in our group we are making extensive use of this library (we develop an iOS application). Recently, I encountered an interesting crash due to an unobserved exception. An unobserved exception is basically an exception thrown from a task, which no other entity (be it the caller or some later continuation) observed (typically by calling task::wait or task::get). In PPLX, such exceptions crash the process (in .NET they used to, and still may depending on configuration).

When an unobserved PPLX exception occurs, the debugger will break in the following location inside pplxtasks.h:

// If you are trapped here, it means an exception thrown in task chain didn’t get handled.
// Please add task-based continuation to handle all exceptions coming from tasks.
// this->_M_stackTrace keeps the creation callstack of the task generates this exception.
_REPORT_PPLTASK_UNOBSERVED_EXCEPTION(); // <– debugger will break here

Your mileage may vary, but I wasn’t able to inspect the _M_stackTrace variable in XCode’s debugger.

However, using the lldb console (you can bring it up with ⌘+Shift+C) I was able to inspect its value:

(lldb) expr _M_stackTrace

Here is a sample output:

(pplx::details::_TaskCreationCallstack) $1 = {
_M_SingleFrame = 0x00c0935d
_M_frames = size=0 {}
}

In this case, the frame of interest is stored in _M_SingleFrame (otherwise the list of frames would have been stored in _M_frames and _M_SingleFrame would have been null). Of course, “0x00c0935d” is not a terribly useful piece of data for root cause analysis, and I wasn’t even sure what that address represented! Fortunately, following macro in the same header clarified that mystery:

#define _CAPTURE_CALLSTACK() ::pplx::details::_TaskCreationCallstack::_CaptureSingleFrameCallstack(_ReturnAddress())

So now we know it is a return address, pointing to the code creating the offending exception. Fortunately, lldb can resolve that address to source:

(lldb) so l -a 0x00c0935d

Sample output:

/Users/ohads/Library/Developer/Xcode/DerivedData/ios-gdhrlqjldqgqktgtpdnpssahaqme/Build/Products/OurApp`pplx::task<void> pplx::task_from_exception<void, std::exception>(std::exception, pplx::task_options const&) + 62 at …

It’s not perfect, but it should be enough to narrow the search down (in this case, there was only one place in our code using task_from_exception).

Should you use std::string, std::u16string, or std::u32string?

C++11 introduced a couple of new string classes on top of std::string:

  1. u16string
  2. u32string

“Finally”, you must think, “C++ has addressed the sorry state of Unicode development in portable code! All I have to do is choose one of these classes and I’m all set!”.

Well, you’d might want to rethink that. To see why, let’s take a look at some definitions:

typedef basic_string<char> string;
typedef basic_string<char16_t> u16string;
typedef basic_string<char32_t> u32string;

As you can see, they all use the same exact template class. In other words, there is nothing Unicode-aware, or anything special at all for that matter, with the new classes. You don’t get “Unicode for free” or anything like that. We do see however an important difference between them – each class uses a different type as an underlying “character”.

Why do I say “character” with double quotes? Well, when used correctly, these underlying data types should actually represent code units (minimal Unicode encoding blocks) – not characters! For example, suppose you have a UTF-8 encoded std::string containing the Hebrew word “שלום”. Since Hebrew requires two bytes per character, the string will actually contain 8 char “characters” – not 4!

And this is not only true for variable length encoding such as UTF-8 (and indeed, UTF-16). Suppose your UTF-32 encoded std::u32string contains the grapheme cluster (what we normally think of as a “character”) ў. That cluster is actually a combination of the Cyrillic у character with the Breve diacritic (which is a combining code point), so your string will actually contain 2 char32_t “characters” – not 1!

In other words, these strings should really be thought of as sequences of bytes, where each string type is more suitable for a different Unicode encoding:

  • std::string is suitable for UTF-8
  • std::u16string is suitable for UTF-16
  • std::u32string is suitable for UTF-32

Unfortunately, after all this talk we’re back to square one – what string class should we use? Well, since we now understand this is a question of encoding, the question becomes what encoding we should use. Fortunately, even though this is somewhat of a religious war, “the internet” has all but declared UTF-8 as the winner. Here’s what renowned Perl/Unicode expert Tom Christiansen had to say about UTF-16 (emphasis mine):

I yesterday just found a bug in the Java core String class’s equalsIgnoreCase method (also others in the string class) that would never have been there had Java used either UTF-8 or UTF-32. There are millions of these sleeping bombshells in any code that uses UTF-16, and I am sick and tired of them. UTF-16 is a vicious pox that plagues our software with insidious bugs forever and ever. It is clearly harmful, and should be deprecated and banned.

Other experts, such as the author of Boost.Locale, have a similar view. The key arguments follow (for many more see the links above):

  1. Most people who work with UTF-16 assume it is a fixed-width encoding (2 bytes per code point). It is not (and even if it were, like we already saw code points are not characters). This can be a source of hard to find bugs that may very well creep in to production and only occur when some Korean guy uses characters outside the Basic Multilingual Plane (BMP) to spell his name. In UTF-8 these things will pop up far sooner, as you’ll be running into multi-byte code points very quickly (e.g. Arabic).
  2. UTF-16 is not ASCII backward-compliant. UTF-8 is, since any ASCII string can be encoded the same (i.e. have the same bytes) in UTF-8 (I say can because in Unicode there may be multiple byte sequences that define the exact same grapheme clusters – I’m not actually sure if there could be different forms for the same ASCII string but disclaimers such as these are usually due when dealing with Unicode:) )
  3. UTF-16 has endianness issues. UTF-8 is endianness independent.
  4. UTF-8 favors efficiency for English letters and other ASCII characters (one byte per character). Since a lot of strings are inherently English (code, xml, etc.) this tradeoff makes sense in most scenarios.
  5. The World Wide Web is almost universally UTF-8.

So now that we know what string class we should use (std::string) and what encoding we should use with it (UTF-8), you may be wondering how we should deal with these beasts. For example – how do we count grapheme clusters?

Unfortunately, that question depends on your use case and can be extremely complex. A couple of good places to start would be UTF8-CPP and Boost.Locale. Good luck 🙂