FOSDEM talk

I will upload the slides soon.

[Update] Slides are up here at slideshare.net

QtWebKit goes Mobile

Repost of Jesus' post, that I helped write.

There is a lot of effort being put into QtWebKit in order to make it attractive on the mobile front.

Among a tons of bug fixes and good performance improvements there are also lots of new features being developed, mainly geared toward mobile deployment.

The goal with this tutorial is to help you understand some of these new features and how to make the best of them. Or said in other words; how to create a good mobile web view that can be used on touch devices.

First we should make it clear that QGraphicsWebView is the way forward, so if you want to target mobile devices, it is bye bye QWebView. Why is that? Well, the QWebView is based on the QWidget system, thus it cannot easily support rotation, overlays, hardware accelerated compositing and tiling. If you need a QWidget anyway, you can always construct a QGraphicsView (which is a QWidget) with a QGraphicsWebView inside. This is more or less what we will start with.

Let's start with the most simple QGraphicsWebView based "browser" ever:

int main(int argc, char **argv)
{
   QApplication app(argc, argv);
   const int width = 640;
   const int height = 480;

   QGraphicsScene scene;

   QGraphicsView view(&scene);
   view.setFrameShape(QFrame::NoFrame);
   view.setVerticalScrollBarPolicy(Qt::ScrollBarAlwaysOff);
   view.setHorizontalScrollBarPolicy(Qt::ScrollBarAlwaysOff);

   QGraphicsWebView webview;
   webview.resize(width, height);
   webview.load(QUrl("http://www.wouwlabs.com/blogs/jeez"));

   scene.addItem(&webview);
   view.resize(width, height);
   view.show();

   return app.exec();
}

Here we just bootstrap a QGraphicsView application and add a QGraphicsWebView to the scene.

It might seem a bit useless as you can only navigate through one website, but it serves well as a simple example. Notice that I'm disabling the scrollbars on the graphics view because QtWebKit handles scrolling and scrollbars automatically. This is due to scrolling optimizations and due to the fact that web authors can interact with the scrollbars for instance style them differently.

On touch-based mobile devices a feature known as tiling is often used. It is used due to the interaction model (touch) as well as a scrolling "optimization". With this optimization we will have to deal with scrolling ourselves and we thus will have to say good bye to the scrollbar styling. Not a big thing, as mobile browsers usually do not even show scrollbars, but use scroll indicators instead.

Tiling basically means that the contents of the viewport is separated into a grid of tiles, so that when you update some area, instead of just updating the area you actually update the whole tile. This gives a few advantages for scrolling as when you scroll you do not need to repaint the new visible area for each scroll step, as you update a row of tiles each time; tiles that are often only partly on the screen. This minimized all the paint calls that we have to do and makes it possible to make nicely kinetic scrolling a possibility.

Loading, layouting etc are blocking operations. Though barely noticeable on a Desktop machines, these operations can block for a long time on a mobile device, letting the user believe the application has become unresponsive and died. Scrolling which is done by using fingers will also stall and give a bad user experience.

One way to over come this issue, is to do all loading, layouting and painting (basically all non-UI related work) in another thread or process, and just blit the result from the web process/thread to the UI. When using tiles, you can blit any tile available when scrolling. When no tile is available you can show a checkerboard tile instead, not letting the scrolling wait for the tiles to be updated. This results in a responsive interface, with the only disadvantage that you from time to time might see checkerboard tiles.

Tiles also helps with zooming. Repainting at each zoom level change during a zoom animation is basically impossible on a mobile device (or desktop for that sake) and thus with tiling, you can stop the tiles from being updates and just scale the already existing tiles, and then at the end of the animation update tiles on top of the scaled ones.

For now we will ignore the blocking issue and concentrate on the tiling and the interaction model.

Resize to contents

When using tiling, we basically want the QGraphicsWebView to act as our contents, as it supports tiling a.o. things. In order for this we need to make it resize itself to the size of its contents. For this we will use QGraphicsWebView::resizesToContents.

From Qt 4.7 documentation: "If this property is set, the QGraphicsWebView will automatically change its size to match the size of the main frame contents. As a result the top level frame will never have scrollbars. It will also make CSS fixed positioning to behave like absolute positioning with elements positioned relative to the document instead of the viewport."

This setting, thus, removes the scrollbars for us on the main frame and makes our QGraphicsWebView resize itself to the size of its content.

Enabling it, is as easy as:

    webview.setResizesToContents(true); 

Qt 4.7 docs also says: "This property should be used in conjunction with the QWebPage::preferredContentsSize property. If not explicitly set, the preferredContentsSize is automatically set to a reasonable value."

If we are going to expand our mobile web view to the size of the contents of its contained page, then that is going to make the view a lot bigger that what can fit on the device's screen!

Using a view as the window to the contents

The idea is to have a custom widget which has a QGraphicsWebView as a class member. Remember that the QGraphicsWebView will be as big as its content's size, so this custom widget will serve as a window, as a viewport.

There is not much more to say here, and the following code snippet illustrates it well:

class MobileWebView : public QGraphicsWidget
{
    Q_OBJECT

public:
    MobileWebView(QGraphicsItem *parent = 0);
    ~MobileWebView();

    bool mousePress(const QPoint &value);
    void mouseMove(const QPoint &value);
    void mouseRelease(const QPoint &value);

private:
    QGraphicsWebView *webView;
}; 

In order to properly handle mouse events you must install an event filter on web view or stack it behind its parent object (search for QGraphicsItem::ItemStacksBehindParent). By doing this the mouse events will reach a MobileWebView instance before they reach the member QGraphicsWebView.

Keep in mind that you'll need to add some logic in order to distinguish different mouse events and gestures like a single click, double click, click-and-pan, etc. That is left as an exercise to the reader.

Also keep in mind that as stated earlier, scrolling will have to be implemented manually, just as zoom etc.

Adjusting how contents is laid out

When testing the above on a device, you will notice that many pages do not layout very nicely. In particular the width is larger than the width of the device!

The way web contents is laid out, is that the first the viewport width is used for fitting the contents. If the contents doesn't fit due to non-flexible element with a width larger than the viewport width, the min width possible will be used. As most pages are written with a desktop browser in mind, that makes only very few sites fit into the width of a mobile device.

QtWebKit has a way to force a layout to a given width/height. What really matters here is the width. If you layout a size to a given width, it will get that width and images etc might get cut of. The width/height is also used for laying out fixed elements, but when we resize the QGraphicsWebView to the size of the contents, fixed elements will not be relative to the view, which is the behaviour found on most mobile browsers.

From Qt 4.7 documentation: "If this property is set to a valid size, it is used to lay out the page."

We saw that this property is automatically set to a reasonable value when using QGraphicsWebView::resizesToContents.

As you can imaging, laying out with a smaller viewport can cause pages to break, and as thus, a default value has been chosen so that it almost breaks no pages while still making the sites fit. This value is 960x800.

If the device have a bigger resolution, this value can be changed using:

    webview.page()->setPreferredContentsSize(QSize(desiredWidth, desiredHeight)); 

You can play around with this and find your own magic number, but let's stick to this 960px wide for now.

The 'viewport' meta tag

As some sites do not work with 960 or want to have control on how the page is laid out, QtWebKit as well as Android, Firefox Mobile and the iPhone Safari supports a meta tag called viewport.

This one also deserves a whole blog post for itself. For now let's just say that this is a meta tag that Apple came up with to make a web page capable of "telling" the browser how it wants to be shown.

More information: http://developer.apple.com/safari/library/documentation/appleapplications/reference/safarihtmlref/articles/metatags.html and http://developer.apple.com/safari/library/documentation/appleapplications/reference/safariwebcontent/usingtheviewport/usingtheviewport.html

In QtWebKit trunk we already have support for this with a nice API. You must connect the signal from QWebPage::viewportChangeRequested(const ViewportHints& hints) to a slot of your mobile web view and use what is provided by QWebPage::ViewportHints to updated your viewport size, scale range, and so on.

This can be tricky and that's why I'm not going deeper on it right now. Since I know you are curious about it I'll leave you with one more exercise! So try to understand how the guys from MicroB and Firefox Mobile dealt with this:

http://hacks.mozilla.org/2010/05/upcoming-changes-to-the-viewport-meta-tag-for-firefox-mobile

Enabling the tiling

We haven't actually enabled tiling yet, so lets go ahead and do that. That is very simple as it is basically a setting:

    QWebSettings::globalSettings()->setAttribute(QWebSettings::TiledBackingStoreEnabled, true); 

Voila! Mind that if you are going to add animations to your zoom/scale or want to implement a fancy kinetic scrolling you might want to take a look at QGraphicsWebView::setTiledBackingStoreFrozen. With this you can avoid updates to your tiles during an animation, for instance.

Avoiding scrollable sub elements

One big issue with the above is that, iframes and sites using frames can contain scrollable sub elements. That doesn't work well with the touch interaction model, as you want a finger swipe to scroll the whole page and not end up just scrolling a sub frame. Most mobile browser work around this by enabling something called frame flattening.

Going straight to the point:

    QWebSettings::globalSettings()->setAttribute(QWebSettings::FrameFlatteningEnable, true); 

This will make all frames from a web page expand themselves to the size of their contents, keeping us free of scrollable subareas.

New Qt WebKit nearing release

So, for the first time, we are going to release Qt WebKit as a separate project. This does not mean that we will stop releasing it as part of Qt, but that if you are stuck on an older Qt 4.6 release, it is still possible to use a newer Qt WebKit. In the future this also allows us to release more often, independently of Qt.

The last half year we have been hard at work on enabling the use of Qt WebKit on mobile devices, while still supporting and maintaining our brilliant support of Desktop applications. Targeting mobile devices is an ongoing challenge due to other interaction modes, smaller screen sizes and less powerful processor.

We have been growing our teams and our responsibilities and it has been an interesting, if not tiring, half a year :-) Locally, we have been growing our Brazilian core team to 5 people, plus creating some new teams working on related projects. What a rush! :-) We now have teams in at least Norway, Finland, Brazil and Boston working directly on WebKit.

So what have our distributed teams been up to?

Well, Qt WebKit has always felt reasonable fast on the Desktop. Not top in the class, but working quite well. On mobile devices the story was very different; dog slow, barely usable! Lots of time has gone into improving that situation by doing profiling and hunting down performance bottlenecks. Improvements that also help the Desktop case. We are not done yet, but our upcoming QtWebKit 2.0 release is a major improvement. Combining QtWebKit 2.0 with the upcoming Qt 4.7, are you will even get a few extra optimizations for free.

Sometimes we had to make more drastic changes, such as using tiling for archiving good scrolling performance. Tiling basically means that you don't paint directly to the screen, but instead to surfaces (which then can actually benefit from hardware acceleration). When scrolling you then move/blit these tiles. The standard way of doing scrolling, is that you copy an area and move that area, but you always have to paint the new area visible. With tiling you normally update a few more row of tiles than those visible on the screen, so when moving you do not always need to repaint. Repainting can actually mostly be done when you are idle.

We also experimented with rendering the tiles in another process, so that scrolling would never be blocked by loading or painting, but with the downside than when there is no tile painted, a checkerboard pattern would be shown, very similarly to the iPhone. Getting this right is a lot of work and with the announcement of WebKit2, we decided to postpone it for now and later concentrate on WebKit2 instead, as that will bring similar benefits, a.o.

Tiling helps with the mobile interaction model, as it is quite fast to zoom. Zooming basically scaled the tiles and then re-renders, which can be done with a smooth animation. Unfortunately, just as on the iPhone, we cannot support fixed elements, so if page authors really need that, they will have to go for CSS3 Transformations which with accelerated compositing will be painted as another layer on top of the tiles.

Which brings me to the next feature: accelerated compositing. Some of the new CSS3 features such as animation and transformations were created with hardware acceleration in mind, for instance the CSS 3 transformation live in another coordinate system, and thus animating a transformation change does not change any DOM values such as for instance 'left'. The idea is that the elements being transformed are painted as a separate layer, composited on top of the normal web contents. On the iPhone, this is implemented using CoreAnimation, I believe, and is thus, tied to hardware support. For Qt, we implemented it on top of out Graphics View system, which means that even when the Graphics System is not being accelerated by hardware, we will archive better performance due to caching of the painted elements. Pretty cool stuff. Unfortunately, just as with tiling, it is only possible to take advantage of this new feature if you use the QGraphicsWebView and not the older brother, QWebView.

You can imagine that most web sites were written for a width a lot larger than what you find on most phones, so either you can show the page in its original size and you will have to scroll a lot, or you can zoom it out to fit within the width of the page. The latter makes it impossible to read the text, so instead of laying out to the size of the width, we have new API (actually originally introduced in Qt 4.6) that makes it possible to lay out the contents as it was having a different viewport. The iPhone and Android does the same, using a value between 800 and 1008 pixels (980 is always default on the iPhone, where as Android varies). Laying out with such a width makes the site's contents still readable most of the time, without breaking the layout. To give the page authors a bit more control over this, Apple introduced a meta tag called "viewport", which is now supported by most mobile browsers, including Mozilla Fennec. This tag allows the site author to specify which width should be used for laying out the contents as well as which zoom factor should be used, and within which limits.

As touch is becoming standard on cell phones, we also had to make a few other adaptations. For instance sites with frame sets are mostly useless due to various scrolling sub elements; the same count for sites with iframes. What most mobile browsers do to fix this, is to apply frame flattening, which we now also supports/ Originally a feature invented by Nokia, but first really being put to good use on the iPhone and on Android.

We didn't leave out non-touch phones either, and as such implemented spatial navigation (aka. keyboard navigation), which is not only useful for cellphones with keyboard only, but also by tv's and set-top boxes.

Not everything has been related to mobile though, and as such our teams have been working on implementing new HTML5 elements such as the Progress element and concentrating on some CSS features and DOM features, such as DOM Level 3 CustomEvents, 'view-mode' Media Feature, as well as some features from CSSOM Views making it possible to listen to media and media feature changes; very useful for creating Web applications.

Wow, it has been a loooong half year it seems! Now time to prepare for our local release parties! :-)

Bossa Conference

So, once more I'm in the Amazon, but this time not for any fancy jungle trip, but instead for the great Bossa Conference! The conference has been really great, with a focus a bit different that other conferences as the focus is on socializing, getting to know people, and actually getting stuff done.

It was really nice to meet my friends from Qt again, as well as meet my fellow WebKit hacker, Ariya Hidayat, who has since moved on to Qualcomm. The conference spotted a lot of technical talks about everything from audio details, bluetooth to now we should develop UI widgets in the near future.

This year is the first year that I did any presentation, so I started out by doing two :-) Here you have the first one, enjoy.

Connecting Technology for Great Experiences - How does QML and Web fit together?

View more presentations from kchristi.

Nokia WebKit Code Camp

So last week Nokia ASF hosted a Qt WebKit Code Camp in Wiesbaden, Germany. It was the first time most of the people working on - or using WebKit inside Nokia, got together and met face-to-face. It is obvious that we have a great bunch of clever people, but also that we still have a lot of work to do before out Qt port is up to the level of the Google and Apple ports, but don'tworry, we are definitely getting there!

Before the event, I had a look at adding tiling support to see if it would actually bring noticable performance improvements. I did a small presentation about it, which you can find below. [UPDATE] The implementation is an experiment to avoid doing unnecessary calls into WebCore and is only meant as input for other people working on adding tiling support to WebKit. I would like to thanks ProFusion, Antti Koivisto and Benjamin Poulain for fruitful discussions who helped to the current implementation.

Considering Tiling for a better User Interface

View more documents from kchristi.

Having working on NPAPI plugin support before, I can tell you it is not a very beautiful part of a browser. Some time ago, we were so lucky that Girish joined us and implemented support for windowless plugins. Now he has even gone one step further and written a huge blog post about it, something that is definately worth checking out. You can find it here:

http://blog.forwardbias.in/2009/12/flash-in-qgraphicsview.html

Also, join me in congratulating him in becoming an official WebKit committer!

At the office we are starting to get into Christmas mood! and friday I'm heading off to Europe to celebrate the Christmas holidays with my family and girlfriend.

Anselmo just took this nice picture from our office:

Merry Christmas to you all!

The world just became a bit cuter

Today is an important day for me, as Qt 4.6 has been released; the first release that I have been deeply involved in.

Actually, I can say that my employer Nokia Technology Institute, INdT, has done a great job on Qt 4.6, as we have contributed to various areas:

• QGraphicsAnchorLayouts: My ex-team implemented a new layout together with Jan Arve, where one can anchor widgets together in the Graphics View. It is extremely flexible, as the layout has no pre-defined concept of how should the items be arranged. The final position of each item is totally dependent on how you set the anchors.
  
  
• Qt Mobile Demos: Three great demos, showing off the power and flexibility of the Qt 4.6 development framework. This video says it all:
  
  
  A cool video by my co-worker Ian Andrade
  
  
• PySide: A Python binding for Qt 4.6, that currently works on Linux, Maemo and Mac OS X, with Windows on the way.
  
  
• QtWebKit: Great works has happened on the WebKit front. Personally I have worked on plugin issues, a new QGraphics-based widget, testing framework, API reviewing, DOM access from C++, among other things. It has been a crazy 9 months since 4.5, and it is amazing how much we have got done.
  
  For more information on the WebKit work, checkout my Maemo Summit presentation, but keep in mind that the PluginDatabase support has been postponed for the next release.
  
  Developments in the Qt WebKit Integration

  View more documents from Nokia, Qt Development Frameworks.
  
  

Please enjoy Qt 4.6! Download it today!

There we Go.

Last week I visited Google for the first time. I even passed by Rob Pikes office...

Today I'm using the language he and his team just created. Lots of inspirations from C, python, limbo and friends, and I actually had fun playing around with it.

1 package main
2
3 import "fmt"
4
5 func PrintAll(x interface{}) {
6     var str string;
7     switch value := x.(type) {
8         case bool:
9             if value {
10                 str = "True"
11             } else {
12                 str = "False"
13             }
14         case string:
15             str = value
16         default:
17             return
18     }
19     fmt.Printf(str + "\n")
20 }
21
22
23 func main() {
24     PrintAll("Hello world");
25     PrintAll(true);
26     PrintAll(false);
27 }

Now look at that :-) Hello World, here we Go.

$ 6g test.go && 6l test.6 && ./6.out
Hello world
True
False

PS: "!" replaced with ".", emphasising that I'm talking about "Go" and not the "Go!" language.