{"id":2081,"date":"2012-03-20T21:57:17","date_gmt":"2012-03-20T21:57:17","guid":{"rendered":"http:\/\/dalelane.co.uk\/blog\/?p=2081"},"modified":"2012-03-20T21:59:37","modified_gmt":"2012-03-20T21:59:37","slug":"avoiding-my-lucene-toomanyclauses-exceptions","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=2081","title":{"rendered":"Avoiding my Lucene TooManyClauses exceptions"},"content":{"rendered":"

Before I start, I should point out that I’m not a Lucene expert. This post isn’t a definitive “you should do things this way” commandment from a Lucene mage. Think of it more as “I had this problem, and this seemed to work for me. I’m sharing it in case it helps you, too”.<\/em><\/p>\n

I’m using Lucene<\/a> to implement searches. Recently, as my Lucene index has grown (a lot), I was getting a lot of these errors when I tried to do a search:<\/p>\n

org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024\r\n    at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:163)\r\n    at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:154)\r\n    at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:63)\r\n    at org.apache.lucene.search.WildcardQuery.rewrite(WildcardQuery.java:54)\r\n    at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:383)\r\n    at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:383)\r\n    at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:162)\r\n    at org.apache.lucene.search.Query.weight(Query.java:94)\r\n    at org.apache.lucene.search.Searcher.createWeight(Searcher.java:185)\r\n    at org.apache.lucene.search.Searcher.search(Searcher.java:86)<\/pre>\n
I’m guessing that TooManyClauses<\/code> is a common problem for people getting going with Lucene. <\/p>\n
It’s mentioned in the FAQ<\/a>, and there are a few StackOverflow threads<\/a> around about it. <\/p>\n
But I couldn’t find a straightforward “you need to follow these steps to fix it” post anywhere, so I’ll add my experience here. <\/p>\n
Why does this exception happen?<\/strong><\/p>\n
To borrow the example from the Lucene FAQ, in some situations Lucene expands your search query before running it.<\/p>\n
If you have an index with:<\/p>\n
     car\r\n     cars\r\n     cat\r\n     catalogue\r\n     caterpillar\r\n     delta\r\n     doctor\r\n     dogs\r\n     dominos<\/pre>\n
And you search for: ca*<\/code>
\nIt will essentially be expanded to “car OR cars OR cat OR catalogue OR caterpillar<\/code>” (removing the wildcard). <\/p>\n
And if you search for: *<\/code>
\nIt will be expanded to: “car OR cars OR cat OR catalogue OR caterpillar OR delta OR doctor OR dogs OR dominos<\/code>“.
\nA query with nine clauses. <\/p>\n
As the index grows, the number of clauses in these wildcard queries increases. If it increases to the point where search queries end up with more than the default 1024 clauses in them, then searches start throwing TooManyClauses<\/code> exceptions. <\/p>\n
Why was I getting this exception?<\/strong><\/p>\n
I had some Java code that was taking user input, and using it to build a Lucene query. The queries where I was ending up with TooManyClauses<\/code> exceptions were where I was trying get all items except a few named entities. <\/p>\n
For example:
\n-id:12345 -id:23456 -id:34567 -id:45678 *<\/code><\/p>\n
In other words, “get me everything except the items with IDs 12345, 23456, 34567 and 45678”<\/p>\n
I was building this using:<\/p>\n
String queryString = \"-id:12345 -id:23456 -id:34567 -id:45678 *\";\r\nQuery luceneQuery = queryParser.parse(queryString);\r\nTopDocs hits = indexSearcher.search(luceneQuery, null, start, sort);<\/pre>\n
This worked fine with reasonably small indices. However, because of the wildcard behaviour described above, this is a terribly inefficient way to do this. <\/p>\n
The wildcard *<\/code> was being expanded to every term in the index, making for a search query that started with “-id:12345 -id:23456 -id:34567 -id:45678<\/code>” and essentially continued with the thousands and thousands of terms in the rest of the index. <\/p>\n
Avoiding the TooManyClauses exception<\/strong><\/p>\n
The Lucene FAQ<\/a> outlines the two main approaches to avoid the TooManyClauses exception. <\/p>\n