Writing an offline wiki client

Friday was the seventh IBM Hack Day, and I again got the chance to spend a day playing with some random ideas.

As Hack Days go, I had a surprisingly productive day! I had four ideas on the day:

  • two mobile hacks (both of which I wrote a chunk of code for),
  • a twitter hack (which never got off the scribbled diagram stage, but it’s an idea I definitely want to come back to), and
  • a hack to extend an IBM product (which I created an alpha version of)

In this post, I’ll describe what I did for the last of these ideas: writing a client app for the wiki that comes with IBM’s Lotus Connections.

The idea

In the same way that I am writing this post in an offline blogging client, I wanted the same for using wikis: read and make changes to a wiki while offline, with changes uploaded to the online wiki the next time you are online.

This wasn’t a new idea. In fact, I tried it at IBM HackDay 4 back in 2007 but the wiki we used at work at the time had no API access for retrieving or updating wiki pages. So I sort of gave up and forgot about the idea.

But now I use Lotus Connections wikis at work. And Lotus Connections does have an API – an AtomPub API that gives you feeds to know when pages are changed, and a way to publish changes.

So I decided to revisit the idea.

The “finished” (ish) hack

It’s still very rough around the edges (this was a HackDay – I wrote the client code in under a day!) but it already shows the basic idea.

Offline wiki client

The top left view shows the list of your wikis.

Clicking on this fills the list below – a list of pages in the selected wiki. Clicking on a page in that list opens the contents of the page in the main view on the right.

There are a few design decisions perhaps worth mentioning…

Java – for cross-platform

I thought it’d be good to write something that (might?) run on Windows, Linux or Mac, so I went with Java. (I did consider basing the hack on my .NET personal wiki app, but I wanted to avoid a Windows-only solution.)

SQLite – for storing pages offline

I did consider using Derby, as this seems to be Sun’s preferred embeddable database of choice for Java apps. But in the end, I still went with SQLite because I’ve used it before and quite like it.

Yeah… should probably have thought of a more technical rationale. But for the purposes of a Hack Day, I didn’t really care – just wanted something I could cache text in!

MQTT – for being useful while online

At a starting point, I could make something which just sucks in a copy of a wiki, so you can read and edit it while you’re offline.

But I was feeling adventurous on Friday. I wanted more than that. I wanted the client to still be useful while you are online, too.

The main improvement is that the app is (or should be!) quicker to open pages than the web-based version.

But to make it a compelling online client, it needed to be able to keep up-to-date.

If someone else creates a page in a wiki while you are online, it should appear in your list of pages automatically. If a page is edited online, the client app should receive this in the background and download the updated page.

So if you use the client app while online, it should just show you up-to-date (ish?) pages as if you were using the web interface.

(Sorry, I suck at video editing, so this video demo is longer than it really needs to be!)

First idea: polling.

The client app could just poll the Lotus Connections API to know when things change.

Bad idea.

On the (okay, remote!) chance that this app gets used by a large number of people, I’m guessing the server admins wouldn’t appreciate thousands of clients hammering their poor server every few minutes.

And infrequent polling (e.g. every 30 minutes) wouldn’t give me the responsiveness I wanted.

Second idea: compromise.

Write one app that polls the API reasonablly frequently (not too frequently, server admins – honest!) and put that on a server somewhere. It publishes notifications when there are changes. The wiki client apps can subscribe to receive these notifications.

Multiple (thousands of?) clients get the client GUI benefits of frequently polling, for the cost of only one app polling the Lotus Connections server.

My server app sucks in the ATOM feed for modified wikis, and uses MQTT to publish any changes. It only publishes the name and modified time for each change – just enough to tell the wiki client apps that they need to download the page again.

But I didn’t want all the (thousands of – am I being optimistic? 🙂 ) wiki client apps to all instantly then hammer the Lotus Connections server to download the modified page at the same time.

Instead, on receiving the notification MQTT message, the wiki client app marks the page as “dirty” in the SQLite database. It will then periodically download an updated copy of all “dirty” pages using the Lotus Connections API – so I’m sort of rate-limiting myself from within the GUI.

The other plus-side to this approach is that I had most of the code to do the server agent piece already to hand from a previous hackday.

What now?

That’s pretty much what I did on the day. There is more that I want to do on the app, such as:

  • Add something to the GUI to let you enter your username and password – at the moment, my username and password is hard-coded in there, so it will submit all edits as me!
  • Add support for more functions from the GUI – such as adding or deleting pages
  • Do more to handle conflicts – at the moment, the GUI highlights if a page changes while you have it open if you are online. But there are other conflict scenarios I’m ignoring – like what happens if you change a page while offline, but the original page changes before you come back online.
  • Make the wiki page editor more user-friendly – at the moment, it is a plain text editor for editing HTML. This is mainly because I couldn’t figure out how to read or write wiki markup using the Lotus Connections API. If I can find a rich SWT widget for reading and editing HTML, that might be good.
  • Make the push server agent bit optional – if you have a private wiki, then this approach will only work if you let the user who runs the server polling agent (i.e. me) have read-access to your wiki. So I want to make sure the client will work okay without it.

Even so, I think it’s a decent start 🙂

Tags: , , , , , , ,

7 Responses to “Writing an offline wiki client”

  1. luis benitez says:

    This is amazing!!! Where can I download this from ? I would love to start playing with this…

  2. dale says:

    Thanks!

    I’ll share a jar for the app in the next day or so – the only reason I didn’t do this on Friday is because my username and password is still hard-coded into the app.

    I’ll try and sort that out this evening – add a dialog to let the user provide their credentials, add a table for config values to the SQLite DB, and update the LC API code.

    Once I’ve done that, I’ll put the app somewhere where people can have a play.

    Thanks again, D

  3. Ben F says:

    Yo Dale – would the buffered JMS client (that uses the MQTT protocol) be of any interest? For offline uses.

  4. dale says:

    Ben – It’s an interesting idea – not one that I had thought of.

    At the moment, unsent changes stay in the local SQLite database, with a “local-dirty” flag set to true. The next time the app gets a connection, it does a “SELECT * WHERE localdirty=true” to get all unsent changes, and attempts to submit them.

    I’ll have a think about whether it’d be better to let JMS handle it, though.

  5. Dale,

    This is a really cool Hack. I am really looking forward to trying it out.

    >If I can find a rich SWT widget for reading and editing HTML, that might be good.
    The Lotus Connections team use the FCKeditor for their editing. I don’t personally have experience of this editor, but it is highly recommended. I know it is not a SWT application like you are looking for, but you should be able to use it in conjunctions with a XULBrowser component.

    Brian

  6. Knut says:

    Hi Dale,

    have you already published your code? Looking forward to try it out. Thanks