Configuring Solr 4.0 to index data from a MySQL database

In a previous post, I documented the steps in setting up Solr 4.0 on a Tomcat 7.0.28 instance. In this post, I'll describe the steps involved in configuring the same Solr instance to index data from a MySQL database.

1.Set up another Solr core.

A Solr core holds the configuration details for connecting to a data store as well as the indexed data of the data store.
The simplest way to set up a core is to copy the collection_1 core and re-name the copy 'test_core'
To make Solr aware of this new core, bring up the Solr Admin console at http://localhost:8080/apache-solr-4.0.0/#/
Select the core-admin option and add core and supply the parameters for name, instanceDir, dataDir, config and schema.

As a result of adding this core, the solr.xml file in the root of the solr_home directory will have the following entry

2. Set up the required libraries

Since to index data from the MySQL database, we will be calling a DataImporter, we will need the Solr libraries. The lib directory is available in your downloaded unzipped version of Solr at c:\apache-solr-4.0.0\dist
Copy all jar files within this directory and transfer them to your solr_home. The solr_home directory should look similar to the screenshot below

I could have transferred these jars to my Tomcat/lib and have solr use them from there but I was worried about conflicts with existing jars.
Finally, tell Solr about this lib directory by modifying the solr.xml file and adding the sharedLib="lib" element.
The solr.xml looks like this.

3. Configure your test_core to talk to the database.

To enable this, you'll need to follow the instructions given on http://wiki.apache.org/solr/DIHQuickStart
To summarise, essentially you will :
(a) Modify the solrconfig.xml and add the defintion for a DataImportHandler

  
    data-config.xml
  

(b) Create a data-config.xml file in the same directory as your solrconfig.xml and specify the database connection parameters

Ensure that the name values in the element have corresponding matches in the schema.xml
For example : The value 'description' should exist in the schema.xml

Finally, we are all set up to import records from our MySQL test schema.

4. Select the test_core in the Solr Admin console and click on the DataImport option which should be last option available in the menu.
Clicking on the DataImport option will bring up the DataImporter Interface, select the Clean and Commit checkboxes and click on Execute Import.

When the indexing is complete, you will see a message giving the number of documents added or deleted.
To check if the indexes have been created as expected and we are getting some sensible results back, perform a Search using 'Query' option
available under the 'test_core'

In my case, I searched for the term 'Hyundai' as I knew of one record having that particular value in the column that has been indexed.
And we get 1 record returned.

Setting up Solr 4.0 on Tomcat 7.0.28

Generally every user centric application requires a Search capability. With Google easily leading the Search domain, user expectations are generally high when they come to a web-application and perform a search. If giving user's the Google experience is not possible, consider Apache's Solr  which is a popular, open-source Java based search tool that runs Lucene under hood.

While Solr comes packaged with Jetty and has a basic tutorial, that provides pointers on its different features, setting up the tool to run within an existing Apache Tomcat instance can require a bit of effort as there are several files and folders and it is not immediately clear which files control the configuration of the tool. This post will document the various steps involved in setting up Solr to run with Apache Tomcat 7.0.28 on a Windows server.

Step 1 : Download a copy of Solr and unzip it  to apache-solr-4.0.0.

Step 2: Create new directory which will be the home of Solr c:\solr_home and copy the following files and folders into it.  ( You will find  them within  the unzipped apache-solr-4.0.0 folder).
bin
collection1
solr.xml
zoo.cfg

collection1 is an existing example which we will load up into our Tomcat.Within the directory collection1, you'll find two other folders conf & data. Just verify that these are present.

Now we'll connect Solr to Tomcat.

Step 3:  Copy the apache-solr-4.0.0.war (which you will find in apache-solr-4.0.0\dist)  into your Tomcat web-apps directory.

Step 4:  Finally, the last step. Tell Solr where the existing search configuration files for collection1 reside.
To do this add the location of the Solr Home directory to your Tomcat JAVA_OPTS. I am setting this value in the setenv.bat file and the declaration looks like this :

set JAVA_OPTS=%JAVA_OPTS% -Dfile.encoding=UTF-8 -server -Xms1536m -Xmx1536m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+DisableExplicitGC -Dsolr.solr.home=C:/solr_home

Step 5:  Start Tomcat and browse to your loaded Solr instance at http://localhost:8080/apache-solr-4.0.0/#/ You should see the Solr Admin screen as below :

Solr Admin Console for Collection 1

To check if Solr has been configured and set up correctly, select collection1 in the left hand side bar and select the query option. Enter 'solr' in the textbox labelled 'q' as shown in the screenshot below.

Default Search Interface

On submitting the query, one result is returned by Solr. On clicking on the result link, the record is displayed as an XML.

Search Result for 'Solr'

In the next post, I'll document the steps of integrating Solr with an existing web-application which is running on the same Tomcat instance with a MySQL back-end.

Setting up HIbernate Search on a Maven build

Hibernate Search is an interesting companion of the Hibernate core components. Since most applications with a public facing view invariably require, a search capability, Hibernate Search presents itself as an east to configure and implement, search tool that fits in well with the Hibernate ORM framework, Under the covers, Hibernate Search uses the Apache Lucene search engine and it is possible to write search queries directly targeting Lucene.Since Hibernate Search is annotation driven, it is preferable to have an annotation based implementation of Hibernate Core of your data model.

The following dependency definitions in your pom.xml will add the required Hibernate Search jars to your project.

org.hibernate
hibernate-search 
3.4.2.Final 

Add the following property settings to your Hibernate.cfg.xml file.

org.hibernate.search.store.FSDirectoryProvider


lucene

These setting will direct Hibernate Search to create the Index base and store it on the File System in a directory named Lucene.

This is all you need to add Hibernate Search to your current project setup. Now all you need is to annotate the fields in your domain model that need to be searchable,

The key annotations are as follows:

@Indexed

@Table(name = "User")

public class User implements Serializable

{

@Field(index=Index.TOKENIZED, store=Store.NO)

private String userName;

}

The @Indexed annotation tells Search to Index this particular table.

The @Field(index=Index.TOKENIZED, store=Store.NO) annotations on the attribute definition tells Search to tokenize the field using the default analyzer. The store annotation tells Hibernate not to store the data with the index and this is the default behaviour. Storing data within the index is required is using a Projection.

You can read more about setting up Hibernate Search at Getting Started

Is 'Cuil' really COOL?

Late last month Cuil (pronounced 'Cool') became the latest entrant to the search domain. Launched by an ex-Googler, Cuil is old Irish word for 'Knowledge'. Its claim of claim of being the 'world's biggest search engine' and its approach in indexing information based on relavance and content has generated a lot of interest and hype.Barely a week old (It was launched on July 28, 2008), it is seen by some as a threat to Google's monopoly over the search market. WOW!!

In a note on its website, Cuil says that it 'searches more pages on the Web than anyone else---three times as many as Google and ten times as many as Microsoft.' and 'Rather than rely on superficial popularity metrics, Cuil searches for and ranks pages based on their content and relevance.'

Hmm..interesting claims. I decided to test them out. I Googled the word 'Cuil' on Google and 'Cuiled' the same on Cuil. The findings were even more interesting

Cuil gave me 121,578 results. In the first ten results, only the first 2 referred to itself. Google on the other hand (kudos to them for fair play), gave me 2,200,000 (is this figure an approximation?) results. Out of the first 10, 8 results were related to 'Cuil'. Impressive!!

I guess it will take some time before the claims of 'ranking pages based on its relevance can be met' and on a personal note, the black background will have to go.

Lucene.. still needs Google?

While reading up on Lucene, which is advertised as a 'high performance, scalable,powerful and accurate' search engine , I noticed that the search function available on the hosting Apache site was 'Search this site with Google'. Wouldn't a search want to use itself for the very purpose it existed??

Needless to say, it doesn't inspire much confidence in all those who want to use it. Watch this space for more.