Avoiding my Lucene TooManyClauses exceptions

March 20th, 2012

Before I start, I should point out that I’m not a Lucene expert. This post isn’t a definitive “you should do things this way” commandment from a Lucene mage. Think of it more as “I had this problem, and this seemed to work for me. I’m sharing it in case it helps you, too”.

I’m using Lucene to implement searches. Recently, as my Lucene index has grown (a lot), I was getting a lot of these errors when I tried to do a search:

org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
    at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:163)
    at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:154)
    at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:63)
    at org.apache.lucene.search.WildcardQuery.rewrite(WildcardQuery.java:54)
    at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:383)
    at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:383)
    at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:162)
    at org.apache.lucene.search.Query.weight(Query.java:94)
    at org.apache.lucene.search.Searcher.createWeight(Searcher.java:185)
    at org.apache.lucene.search.Searcher.search(Searcher.java:86)

I’m guessing that TooManyClauses is a common problem for people getting going with Lucene.

It’s mentioned in the FAQ, and there are a few StackOverflow threads around about it.

But I couldn’t find a straightforward “you need to follow these steps to fix it” post anywhere, so I’ll add my experience here.

Read the rest of this entry »

Tags: , ,
Posted in code | Comments Closed

Face Movies

March 18th, 2012

I’ve been using Picasa to manage my photos of the kids for years.

I noticed in the changelog for the latest version that it now can create chronological “Face Movies”. These are videos of all the photos of a certain person, arranged in order that they were taken, and zoomed, rotated and cropped to line up the face as best as possible between the pictures.

After a little prodding from Tim, I gave it a try.

Here is (nearly) every photo I’ve ever taken of Grace, run through Picasa’s Face Movie.

It takes two-and-a-half minutes, but I think it’s pretty cool. I love how you can watch her face change, and her hair grow over time.

There are settings you can tweak. For example, you can change the frame rate, or get it to filter out photos taken within a specified amount of time of each other. It means you can make shorter versions – I made a video of just some of Grace’s pics that lasts under a minute.

Read the rest of this entry »

Posted in kids | 1 Comment »

A nice memory

March 16th, 2012

I was reminded of this by my Timehop email this morning. (Incidentally, I’m loving Timehop – thanks to Roo for sharing it).

A year ago today, I had a lousy afternoon. I even tweeted about it.

I didn’t share what happened after, although I possibly should have.

Read the rest of this entry »

Posted in kids | 1 Comment »

Migrating from ActiveMQ to WebSphere MQ

March 13th, 2012

Overview

A side-project I’ve been playing with in the evening: Writing a JMX layer to allow apps written for ActiveMQ to migrate to WebSphere MQ with minimal modifications

Background

This came out of working on something that uses a JMS messaging provider. It uses it internally to allow components to communicate with each other, even when spread across multiple machines.

It uses Apache ActiveMQ – an open-sourced implementation of JMS. I wanted to try and get it working using WebSphere MQ – IBM’s implementation of JMS that I used to work on until five years ago.

As a messaging standard, the fact that both ActiveMQ and WebSphere MQ (WMQ) are JMS providers means that the way it puts and gets messages should just work.

But the JMS standard doesn’t cover administration (how queue managers are created and configured, how they’re started, how queues and topics are created, etc.) or monitoring (getting statistics about how many messages have been put or got, how many messages are on a queue, etc.)

All of this was done in an ActiveMQ-specific ways. This was what needed to be ported if I was going to get this to work with WebSphere MQ.

The project I’m porting is actually a bit of a black box. Rather than make a significant rewrite to get it to go from being ActiveMQ-specific to WMQ-specific, I wanted to see what I could add so that as much of the existing code could just work transparently.

I wanted to write a layer to sit between the ActiveMQ-app and WebSphere MQ, so that the app needn’t realise it’s not talking to the ActiveMQ broker it was written for.

Read the rest of this entry »

Tags: , , ,
Posted in code, ibm | 4 Comments »

ETag header missing in HTTP No Content responses in Internet Explorer

February 26th, 2012

If you’re one of that exceedingly rare breed who regularly check or subscribe to my blog, you probably want to give this post a miss. This one is more for people who find me through Google. A specific solution to a specific, geeky, problem.

Background

First a little scene setting…

Server side

I have a REST API that uses ETags for, amongst other things, concurrency control. That is, the version of an entity is (opaquely) identified by an ETag. You need to specify that ETag when you try and make any changes to that entity. If someone else changes the entity before you do, your ETag won’t match, so your update will fail, and you won’t unintentionally roll-back their change.

The REST API returns no content (HTTP 204) in response to a successful PUT request to edit an entity, and includes the new ETag representing the version of the updated entity.

Client side

I have a Dojo web tool that uses xhr.put to submit edits to the REST API. In order to make further subsequent edits to an entity without reloading the page, it stores the ETag that it gets back in the response header after every PUT.

The problem

In short, Internet Explorer. 🙂

Read the rest of this entry »

Tags: , ,
Posted in code | 2 Comments »

Posting to IBM Connections from WP7

February 24th, 2012

IBM Connections is something we use at work: an internal, intranet-hosted social network for work stuff.

I use stuff like the wikis, file sharing, and bookmarks, quite a lot. But I don’t make status updates as often as I could.

I wonder if that’s because I couldn’t do it from my phone? I know that I certainly started using Facebook a lot more since getting a phone with support for posting to facebook and twitter built-in.

So I set up a way for me to post to Connections from my phone with just one tap on the home screen.

Well, one tap, plus all the taps to actually type the status message… plus another tap on the Send button. But you get the idea.

Read the rest of this entry »

Tags: , ,
Posted in code, ibm | 1 Comment »

A week off for half-term

February 18th, 2012

This week has been school holidays in Hampshire, so I took a week off to entertain the rampaging monsters otherwise known as my children.

As always, it’s been a fun mix of things. With a three year old and a seven year old, the general plan is to try something a bit different every day so they’re not bored by the end of the week.

We did a few arts and craft things. We made our own candles, melting down candle wax chips from Hobbycraft in a saucepan, adding colouring and pouring them into moulds. We did some sewing, making stuff like animal-shaped cushions.

We spent an afternoon mucking about with a camera and a microphone. Faith loves singing into a small USB microphone we’ve got, so we recorded her singing some of her favourite songs. Grace tried making a stop motion animation using her Sylvanian Families. It ended up being 25 seconds long, but between taking all the photos, and recording the voices and sound-effects, it took her ages!

Read the rest of this entry »

Posted in kids, misc | 2 Comments »

Making an evaporograph

February 17th, 2012

It’s still half-term, so time for another random “what can I do with the kids that’s vaguely science-y and only needs stuff that we have around the house”.

We made an evaporograph.

No, I’d never heard of it before, either.

But it was pretty cool which is why I’m blogging it myself instead of leaving Grace to write it up 😉

Read the rest of this entry »

Posted in kids, misc | 2 Comments »

« Older Entries
Newer Entries »