Dynamically Creating Solr Cores From ColdFusion 9

As you may or may not know, Solr allows you to create and maintain several "cores". Each core can have its own configuration, schema and index. This comes in very handy if you maintain several applications, or different divisions of a single applictaion. You can manually create a core on the Solr server itself by creating a folder to contain the config information and index and adding the name of the core to the solr.xml file, but physically accessing the server and manually creating all of these files isn't always an available solution. So what now? Glad you asked.

First, you're going to want to make sure your Solr server is set up to store your created cores properly. Pop open solr.xml in your multicore folder. You'll probably see something like this:


<solr persistent="false">
<cores adminPath="/admin/cores" host="${host:}" hostPort="${jetty.port:}">
<core name="core0" instanceDir="core0" />
<core name="core1" instanceDir="core1" />
</cores>
</solr>

The important part is persistent="false". By changing this to true, we're telling Solr that we want cores we create "on the fly" to be permanent additions to our list of cores. When this is set to false, the cores we dynamically create will be deleted the next time the server is restarted. Set this to true and, if Solr is already running, restart Solr.

Solr makes it very easy to create a new core with a simple http request.

http://{IP.of.Solr.Server}:8983/solr/admin/cores?action=CREATE&name={NameOfNewCore}&instanceDir={DirectoryContainingConfig}&dataDir={WhereToStoreIndex}

Breakdown of parameters: 8983 - The default port that Solr communicates over. This can be changed when Solr is started if needed, but I haven't had to change this myself.
action=CREATE - Tells Solr we're creating a new core
name - What we want to name our new core
instanceDir - This one requires a bit of explaination...

This will be a directory on the server that contains the config and schema information needed for the core. Naturally we're not going to be dynamically generating these files. What I have done for my appication is manually create a core called "Template". In the Template folder lives in the multicore folder and contains the config and schema files set up for my application. If you're creating cores for a specific application, the schema will be set up to match the fields you need. Otherwise, you can create a generic set of fields that will work in a variety of appllications, or use dynamic fields, which are a bit more advanced and I won't get into now. When I create a core, I set my instanceDir to "Template" and it uses the files in this folder as a template for the new core. This method has worked very well for my needs.

dataDir - The folder that will contain the core's index. This folder does not have to already exist. Solr will create it when the core loads.

There are two additional parameters that I typically don't use:
config - the name of the core's config file
schema - the name of the core's schema file

By default, these are named solrconfig.xml and schema.xml. I tend to just stick to the defaults. If you want to name them something else, you'll have to add the parameters when you create your core.

You can create a core from ColdFusion by doing something like this:


<cfscript>
newCoreRequest = new http();
newCoreRequest.setMethod("get");
newCoreRequest.setURL("http://localhost:8983/solr/admin/cores?action=CREATE&name=MyNewCore&instanceDir=Template&dataDir=MyCoreData");
response = newCoreRequest.send().getPrefix();
</cfscript>

The response will contain a structure containing information about the success or the failure of the creation of the new core.

If you don't want to write it yourself, I have good news for you. I got ambitious last night and added two methods to my CFSolrLib for Solr 4.0 GitHub branch. There's a method called checkForCore that checks to see if a core already exists and another called createNewCore that does just that. I've also added an example cfm that shows how to check for a core and then create a new one based on whether or not it already exists. All of this is available on GitHub at https://github.com/VWRacer/cfsolrlib.

In summary, multicore mode is a very useful way to maintain several indexes on one Solr installation. With a simple http call you can create cores as needed from ColdFusion (or any other kind of applictaion for that matter). The new methods added to CFSolrLib will allow you to easily plug this functionality into any existing CF applictaion. As always, I want to thank Shannon Hicks for writing CFSolrLib to begin with. He created the base version and without his initial hard work, I wouldn't have base code to improve upon. If you're working with a 3.X version of Solr, have a look at Shannon's GitHub repository at https://github.com/iotashan/cfsolrlib for the latest code. I haven't tested my version with any of the Solr 3 versions yet. It may very well work, but I know for a fact Shannon's code works with Solr 3.

**EDIT - My modifications to CFSolrLib have been rolled into the original repository. CFSolrLib is now distributed with Solr 4.0.

More Solr goodness to come...

Related Blog Entries

Comments (Comment Moderation is enabled. Your comment will not appear until approved.)
Mark V's Gravatar Jim, at least in the latest version of cfsolrlib that I downloaded last week, the dataDir argument of createNewCore is not required. As a result, if you do not include it, Solr 4.6 attempts to use the instance directory as the data directory, and, because the instance directory is locked while the new core is being created, the creation fails on a lock timeout. For this reason, I believe dataDir needs to be required for this cffunction.
# Posted By Mark V | 1/20/14 1:45 PM
Jim Leether's Gravatar You are correct. The version of CFSolrLib available currently has only been tested through version 4.0, which did not require the dataDir. I'm currently working on an updated version of CFSolrLib designed to work with 4.6 that will also include new functions for adding fields and copyfields to a managed schema and some other new features available since the 4.0 release. I haven't pushed any of my updates to my fork of CFSolrLib yet, but they will be coming soon. These changes will also be part of my presentation at CF.Objective() in May.
# Posted By Jim Leether | 1/20/14 1:52 PM
John's Gravatar A very informative article. Thank you!

Just one thing, I find it really hard to read white text on a black bg. It's your blog, your style. I'm just sayin' :)
# Posted By John | 3/25/14 12:05 PM
Jim Leether's Gravatar Good to know, both that you found things informative and that you had trouble reading it. That definitely matters to me. I'm planning a blog style overhaul soon. Hopefully I won't sacrifice function in the name of fashion.
# Posted By Jim Leether | 5/19/14 12:56 PM
Copyright © 2008 - Jim Leether BlogCFC was created by Raymond Camden. This blog is running version 5.9.1.001. Contact Jim