Everyone who has ever set up a search interface for a client has heard it. "We want it to automatically fill in like Google." It sounds simple enough, but I definitely experienced a bit of a learning curve setting this up. There was a lot of conflicting information out there and there was a lot of trial and error. This is what I finally got to work. There are a couple ways to accomplish this type of UI with Solr, the two most popular being JQuery UI Autocomplete and Bootstrap Type Ahead. Today, we're going to discuss JQuery UI Autocomplete.

The first thing we need to do is set up Solr to create a library of words to make available to fill in our text field. We start by adding a field and field type to our schema.

Schema.xml

When initially setting this up, I found a variety of articles on how to set up the field type to return phrases instead of single words. A lot of the articles described the way it was SUPPOSED to work, but for me it never did. Through some more research and tinkering, I found that using the Standard Tokenizer Factory with the Shingle Filter Factory in the index analyzer did the trick. Also, be sure to use the Remove Duplicates filter in your query analyzer so you don't get duplicate results at query time. Finally, we add a copyField tag to tell Solr to use data put into the "content" field to be used for our dictionary. This occurs at index time. You can change this to whatever field you want your results to come from. For example, if your users will be searching on the "title" field, you'll want to copy title data into the content_autosuggest field. In my case, they're searching for text within indexed documents, so I'm using the content field.

Next, we need to set up a search component and request handler in our solrconfig to handle our Auto Suggest requests. Technically, auto suggest is a spell check component since it's actually taking our keystrokes and suggesting possible alternate spellings to complete the word or phrase we're typing in. We set it up like so.

solrConfig.xml

We name our component "suggest" so we know when we query against this component, we're getting back our auto complete "suggestions". Really you can name it whatever you want, but I found "suggest" to make the most sense. There is a bit of mixed information out there on what to set in the "lookupImpl" attribute. This is the lookup class used to make matches. Some information I have read says to always use the FSTLookup due to it's performance. In my case, after some tinkering, I found TSTLookup to work for me better. Information on the different available classes is available on the Solr Wiki page. In the "field" attribute, we list the name of the field we're using for our auto suggest data and set "buildOnCommit" to true. This will ensure that as new content is indexed and committed, it is made available to the suggester component. You can set this to false to save resources, but you will have to run the build command manually to get any new data into the dictionary. The "storeDir" attribute tells Solr where to build the dictionary file. If you do not specify this attribute, the dictionary will be built and stored in memory. This eats up A LOT of memroy. We then specify the field type used for the auto-suggest data, which we set to text_auto in the schema.xml file above.

The request handler is relatively straightforward. We set up a search handler called "/suggest" and set up some default values. Set the "df" (default field) value to content_autosuggest, the field we use exclusively for auto suggest data. Since our component is a spellcheck component, we also have to set up some default values for the spellchecker. First, setting "spellcheck" to true lets Solr know we are using a spellcheck component. The "spellcheck.dictionary" attribute specifys which dictionary (or spellchecker) we're pulling our results from. We're setting this to the "suggest" component we defined above the request handler. The "onlyMorePopular" attribute is not used for spell check, but is used with the suggester to return results sorted by frequency rather than alphabetically. "Count" is simply how many results to return per request. Setting "collate" to true modifys the query slightly. This just ensures that we get the top results for our search term by ordering them properly. Finally, we tie our request handler to our suggest component by adding it to the "components" section of the request handler.

That's it for the Solr setup. I know that was a lot to take in. There's definitely A LOT of configuration to do and it's easy to make a mistake here or there getting all of the pieces tied together. Take it slow and pay attention to the details. As always if you hit a snag, the guy with the green hair is here to lend a hand.

Now, on to the ColdFusion side of the house.

In the latest release of CFSolrLib, there's a method in cfsolrlib.cfc called "getAutoSuggestResults".

getAutoSuggestResults method:

There are lots of loops in there that are basically building lists of suggestions. The cfc sets up the http call to the suggester request handler and parses the XML that gets returned. I've done a little work so the cfc knows if we're typing our second or third word, the suggester takes that into account when looking for suggestions instead of just looking for the word we're currently typing. It's getting back the top valid results from solr and then alphabetizing the list to make it a little more pleasing to the user's eye. It also does a little error checking to make sure we're getting a valid result back from Solr. If not, it simply returns a blank result rather than throwing back an error to the form and blowing the whole business up in the user's face.

Now onto our form.

First, we include JQuery and JQuery UI to make sure JQuery's autocomplete methods are available. For this example, I just created an input called "keyword" that we will be using to generate our results. In the script block at the top, we're binding our input to our CFC that makes the call to Solr and specifying that we want JSON as our return format. JQuery Autocomplete expects JSON as it's data.

As long as you already have information in your index, Solr will build the dictionary when you start it up. If not, start Solr and index a few things. Since we set buildOnCommit = "true", the items will be added to our dictionary when we commit our changes to the index. You can always manually rebuild your dictionary at any time like so.

You can simplify this further by just typing that URL into a browser to rebuild the dictionary, but this code snippet works well if you want to insert a button or link into an application to rebuild your dictionary on the fly while debugging.

If all went well and you have all of your bits and pieces set up correctly, you should be able to run this in a browser and see results drop down as you begin to type in the input box.

There's a fully functional example of this code, including a properly set up Solr 4.0 instance in the latest CFSolrLib, available on GitHub.

A working example can be viewed at http://jimleether.com/solrexample/autoSuggestExample.cfm

This is definitely a lot of information to take in. If you get it all working on your first try, well kudos to you. When you get everything customized to your application, this is a very powerful tool.

I plan on writing a post about Bootstrap Type Ahead and Solr very soon. Enjoy!

I had a great post written on multicore mode, indexing and searching in multicore mode and doing a distributed search over several cores, but good ole Skype locked up my computer and I lost the entire thing. So instead I decided to touch on this subject.

So we all know Solr is nifty by now. If you've done any in depth reading its capabilities, the ways you can break up and analyze text to make your searches are relevant are endless. Solr uses Apache Tika to parse many types of documents and Tika does a great job extracting content and metadata from a large number of different kinds of files.

Those of you using Solr with your CF applications probably know that it's smart to load Tika on the ColdFusion side and parse a file's content before sending it over to Solr. That way you're not streaming an entire file over HTTP, but simply sending a string. This frees up resources and bandwidth for other things, and is also a lot faster. There is an example of how to use Tika on the ColdFusion side in the index example in the latest CFSolrLib.

In an application I maintain the Solr server and code for, things were humming along just fine until I tried to parse a .docx file. All of the sudden the application chokes and I receive a ColdFusion error. This became a huge thorn in my side and was happening with any of the "newer" Microsoft Office files with file extensions ending in "X" (aka Open XML). They became known around the office as "X-Files". In the stack trace of the error was:

Caused by: java.lang.ClassCastException: org.dom4j.DocumentFactory cannot be cast to org.dom4j.

So what the heck does that mean???

To put it in English, Tika processes Open XML formatted documents (docx, xlsx, pptx, etc) in a different way and as a result, must use a different context class loader. This used to be difficult and required a lot of code, but thanks to Mark Mandel, a switchThreadContextClassLoader method was added to Javaloader that does this for us automatically.

In the application I work on, the file extension is stored in a database, so it's very easy for me to make a comparison and switch the context class loader when needed:

The code above reads the file extension and decides how to process the incoming file based on whether or not it's an Open XML formatted file. If it is an Open XML file, it calls the switchThreadContextClassLoader method and loads a new instance of Tika in the processOpenXMLFile method, where the file content is parsed and returned.

Here's look at the processOpenXMLFile method:

Take note of the tika.setMaxStringLength(-1); setting. By default, Tika will only extract the first 1000 characters from a document. You can set this to as many characters as you want, but setting it to -1 will remove the restriction altogether.

Using the code above will allow your application to handle any Open XML file you want to throw at it, just make sure the file extensions you need are listed in the function. If you need to place the code in more than one location, you could store the list of file extensions in a database or variable, that way you only have to maintain it in one place.

This error was a huge pain for me and I want to thank Mark Mandel and Jeff Coughlin for helping me to flush out the issue. I'm a big fan of "passing along the love", so hopefully this will help someone who encounters this problem flush it out in a timely manner. I wasn't able to find much on the subject, and what I was able to find was more for the Java programmer, which I am not.

Have fun out there and keep on indexing!

As you may or may not know, Solr allows you to create and maintain several "cores". Each core can have its own configuration, schema and index. This comes in very handy if you maintain several applications, or different divisions of a single applictaion. You can manually create a core on the Solr server itself by creating a folder to contain the config information and index and adding the name of the core to the solr.xml file, but physically accessing the server and manually creating all of these files isn't always an available solution. So what now? Glad you asked.

First, you're going to want to make sure your Solr server is set up to store your created cores properly. Pop open solr.xml in your multicore folder. You'll probably see something like this:

The important part is persistent="false". By changing this to true, we're telling Solr that we want cores we create "on the fly" to be permanent additions to our list of cores. When this is set to false, the cores we dynamically create will be deleted the next time the server is restarted. Set this to true and, if Solr is already running, restart Solr.

Solr makes it very easy to create a new core with a simple http request.

http://{IP.of.Solr.Server}:8983/solr/admin/cores?action=CREATE&name={NameOfNewCore}&instanceDir={DirectoryContainingConfig}&dataDir={WhereToStoreIndex}

Breakdown of parameters: 8983 - The default port that Solr communicates over. This can be changed when Solr is started if needed, but I haven't had to change this myself.
action=CREATE - Tells Solr we're creating a new core
name - What we want to name our new core
instanceDir - This one requires a bit of explaination...

This will be a directory on the server that contains the config and schema information needed for the core. Naturally we're not going to be dynamically generating these files. What I have done for my appication is manually create a core called "Template". In the Template folder lives in the multicore folder and contains the config and schema files set up for my application. If you're creating cores for a specific application, the schema will be set up to match the fields you need. Otherwise, you can create a generic set of fields that will work in a variety of appllications, or use dynamic fields, which are a bit more advanced and I won't get into now. When I create a core, I set my instanceDir to "Template" and it uses the files in this folder as a template for the new core. This method has worked very well for my needs.

dataDir - The folder that will contain the core's index. This folder does not have to already exist. Solr will create it when the core loads.

There are two additional parameters that I typically don't use:
config - the name of the core's config file
schema - the name of the core's schema file

By default, these are named solrconfig.xml and schema.xml. I tend to just stick to the defaults. If you want to name them something else, you'll have to add the parameters when you create your core.

You can create a core from ColdFusion by doing something like this:

The response will contain a structure containing information about the success or the failure of the creation of the new core.

If you don't want to write it yourself, I have good news for you. I got ambitious last night and added two methods to my CFSolrLib for Solr 4.0 GitHub branch. There's a method called checkForCore that checks to see if a core already exists and another called createNewCore that does just that. I've also added an example cfm that shows how to check for a core and then create a new one based on whether or not it already exists. All of this is available on GitHub at https://github.com/VWRacer/cfsolrlib.

In summary, multicore mode is a very useful way to maintain several indexes on one Solr installation. With a simple http call you can create cores as needed from ColdFusion (or any other kind of applictaion for that matter). The new methods added to CFSolrLib will allow you to easily plug this functionality into any existing CF applictaion. As always, I want to thank Shannon Hicks for writing CFSolrLib to begin with. He created the base version and without his initial hard work, I wouldn't have base code to improve upon. If you're working with a 3.X version of Solr, have a look at Shannon's GitHub repository at https://github.com/iotashan/cfsolrlib for the latest code. I haven't tested my version with any of the Solr 3 versions yet. It may very well work, but I know for a fact Shannon's code works with Solr 3.

**EDIT - My modifications to CFSolrLib have been rolled into the original repository. CFSolrLib is now distributed with Solr 4.0.

More Solr goodness to come...

For those of you who saw my presentation at CF.Objective(ANZ), you know that I had updated CFSolrLib to work with Solr 4.0. It was a bit buggy and SolrJ was still in Beta, but it worked. When I got home, I forked the original CFSolrLib branch on GitHub and made my changes public. Now that SolrJ has a release version for version 4.0, the final changes have been committed.

I had some challenges getting this all to work and I figured I'd share what I discovered. It will save headaches in the future. I thought I was being a "good little code monkey" by upgrading all of the jar files SolrJ uses to communicate with Solr. Some of the updates were necessary, as quite a few of the methods used in the SolrJ API for previous versions of Solr had beed depricated. After upgrading, however, I ran into a flurry of problems. To make a long story short, what I discovered is that ColdFusion 9 uses slf4j-log4j12-1.5.6.jar. I had upgraded the log4j jar files in CFSolrLib to version 1.6.6. These two versions are imcompatible and cannot communicate with each other. Shiny and new is tempting, but apparently not always the best path.

The error messages in the stack trace were not the most descriptive.

org.slf4j.impl.Log4jLoggerAdapter.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V

I went round and round quite a few times before I figured out which files were causing the issue. I downgraded the log4j files in CFSolrLib down to 1.5.6 and all was well in the CF and Solr world again.

For those of you wishing to tinker with ColdFusion and the standalone Solr 4.0.0 server, you can download the newest version of CFSolrLib from my GitHub at https://github.com/VWRacer/cfsolrlib/. The download includes the most recent version of Javaloader as well as a version of Solr 4.0 set up to work with the example. It also includes examples for indexing, searching, using result highlighting and setting up an auto-complete suggestion as your users type in a search string. I have a pull request in place, so hopefully my changes will become part of the original CFSolrLib repository soon. Documentation on setting up your own custom instance of Solr 4.0 is available on the Apache Solr Wiki page, although I HIGHLY recommend Solr Enterprise Search Server and Solr Cookbook publications available. They were a huge help to me when I was getting spun up. The Cookbook has released a version for 4.0, but the Enterprise Search Server book is still only available for version 3. It's still very valueable, but I expect a version 4 book will be released shortly.

Keep your eyes here for more Solr and ColdFusion examples in the future. I plan on writing up a multicore example very soon.

Recently, I was informed that I was going to take over development on an AIR application that I wrote the original prototype for back in 2007. Back then my skill set was limited and the code definitely had its issues, but I managed. The problem now is that I haven't written a line of MXML or AS3 since then. The other issue is that many of the newer spark components were not available then, and I have no experience with them. The application, which has been in the hands of a few other talented Flex developers in the mean time, is in great shape. I'm taking over where the last developer left off and he left very good notes as to what was being worked on when he stopped. I just needed a refresher to get me going again.

I'm currently about halfway through a three-day Intro to Flex course from Ledet Graphics Training. So far, I'm very impressed. It's being held at Zenith Technologies, who has gone out of their way to make us comfortable at the training facility. The instructor is extremely knowledgeable in his field and is actively developing, which has kept his skills fresh. He was able to show us some real world examples of his work.

My lunch break is over....back to work for now.

I returned late last night from CF.Objective(ANZ) in Melbourne and had a wonderful time. Great sessions, great friends and colleagues and a great overall experience. I especially want to thank Kai Konig, Mark Mandel and Julie Allen, whom without this event would not have been possible.

The keynote was extremely inspiring to the CFML developer. Mark Drew presented on some of the new great features in Railo 4 and showcased future support for the language. I think everyone left with that "warm fuzzy" feeling. The sessions that followed highlighted technologies like Clojure, FW/1, CFBulder, PhoneGap, digital video, using SCRUM efficiently and, of course, Apache Lucene Solr. The information provided in these sessions was invaluable. A very solid group of professionals was represented at CF.Objective(ANZ) and it was a priveledge to present alongside them.

I want to thank all that attended my workshop and my session on integrating Solr with ColdFusion applications. It felt great to pass along my enthusiasm for this technology to others.

For those interested, slides and code samples can be downloaded here.

I know there are a couple questions some of you had that I need to research. I will post the answers to those shortly.

Thank you again and I hope to see you all at a future conference.

Steve Bryant had the great idea of making August 1st "How I Got Started In ColdFusion" Day. He proposed that every blogger write a post about their beginnings in ColdFusion. This is my story.

Back in 2007 (yes, I'm actually quite a noob. Kind of speaks to how easy ColdFusion is to learn) I was holding down a job cutting glass in a restort beach area in Ocean City, MD. I had a strong background with computers, networking and programming in several languages, but since I didn't have a degree in this field, getting a job was a bit difficult. I wasn't really making enough money to support my family. I was driving 90 miles a day to work and back, the price of gas was killing me, my home was in forclosure, my car was days away from getting reposessed and I just generally wanted to go play in traffic. I was having a conversation with my good friend, and now co-worker, Yancy Wharton about the state of my affairs. I was miserable in my current job and my home life certainly wasn't a ray of sunshine. He queried "Do you think you could learn ColdFusion?". I thought about it for a few seconds. Before I could get a word out, Yancy was explaining what ColdFusion was, the benefits of learning the language and that my background in other languages should make it very easy for me to pick up. I agreed to give it a shot, and Yancy went to talk to his employer about picking me up part time.

A few weeks later, I found myself sitting in front of a laptop setting up a work environment. I was still working my full time days at the glass company, but was working another four hours in the evening as a programmer. Initially, I was hired to write a prototype for an offline/online version of the company's application. Thus I began my journey with Apollo, which most of you know now as Adobe AIR. This was my first experience with MXML and Actionscript. I ended up with an application that did what needed to be done, but my lack of programming experience with the language made for some very hard to read code and lengthy files. The application was later rewritten by some other members of our company, who made it into a very impressive product. My first ColdFusion was writing the components to handle the back end of the AIR application.

Four months later, I was hired full time and moved to strictly ColdFusion code. I was amazed at how simple it was to work in CFML. I spent a lot of the time away from work browsiing through tutorials and training videos, but the real CF addiction started when I attended my first CFUnited. When I got to see the industry professionals creating wonderful things and doing these presentations showing how easily they can be done, I was hooked. I became bound and determined that I was going to immerse myself in this and become the best I could be at it.

So, in a sense, ColdFusion saved me. I very quickly got a job making enough money to save my house, catch up on my car payments and start dragging myself out of debt. My mood improved greatly. I was doing something I truly enjoyed and felt secure in my position. I was watching my creations come to life on the screen.

Today, I'm still happily plugging away at ColdFusion. I'm working full time for a government contractor and I have my own small development and consulting company at home. I still play around with the occasional AIR app, but for the most part, ColdFusion is my language of choice. It has brought me into new technologies like JQuery (which I know works with several languages, but it works DAMN well with CFML), which I'm truly enjoying as well. I'm also the Co-Manager for the Eastern Shore of Maryland User Group.

So I'd like to thank everyone who has helped to mold me into the CFDude I am today. Thank you Ray Camden, Ben Nadel, Simon Free, Yancy Wharton, Sean Corfield, Dan Wilson, Mark Drew, Aaron West, Jason Dean, Dee Sadler, Adam Lehman.....the list goes on and on. Sorry if I forgot anyone, the post was going to become an endless list of names eventually. Thank you to all, and thank you ColdFusion.

Well, I finally got rid of my Palm Pre. As you can tell from reading my previous entry, I was very happy with my Pre when I first got it. However, it seems like after a few OS updates, Palm just stopped caring. I hadn't seen a system update in over a year by the time I traded in my phone. It had some design flaws too, one of which caused the screen to crack quite easily. I replaced the phone once because of it and it just ended up happening again.

As for my new HTC EVO 3D, I'm really enjoying it. The first thing that got my attention was the screen size. Compared to the Pre, it's huge. It's slightly narrower than the EVO 4G, which is what my wife chose, but it's still a great display. Android, of course, makes it a breeze to navigate and there are plenty of apps available for Twitter, Facebook, Skype, LinkedIn, etc. So far no compatability issues with any of the apps I have downloaded.

The home screen was extremely thought out. This phone boasts a dual core 1.2 gHz processor, which it makes full use of. EVERYTHING this phone does in animated in some way, and it doesn't lag a bit. Everything from the weather to the time just jumps off the screen, which brings me to the 3D capabilities of the EVO 3D. The downloadable movies and games are VERY impressive. It comes pre-loaded with a demo Spiderman game that's pretty nice. After seeing the Nintendo 3DS I was a bit skeptical as to the ability of this phone, but I have to say I was pleasantly surprised. The built in 3D camera (it uses two cameras to impose two images over one another) isn't as clear as the movies, but you have to consider the difference in the quality bewteen professional movie equipment and the cameras attached to the phone. The pictures can actually be a bit disorienting, but if you focus on them just the right way, they definitely pop. It reminds me of the way I had to relax my eyes to see the old "magic eye" images. Remember those? It's also nice that you can turn off the 3D if you just want to take regular pictures. You can't email or send 3D pictures over SMS. They get saved in their own unique file format.

Overall, I give the EVO 3D a bug thumbs up. Nice size, Android OS, beautiful display and great sound. I'd call it the full package. Great job HTC.

In more ways than one. The first way is obvious, back from a hiatus writing blog posts. The second way is back from CFUnited 2010. The third and final way, being the least fun, is back from a server outage.

The gaps in blog posts are going to end. I think by this point I have enough to write about, I just have to set aside a block of time to sit down and write. I was concerned when I started this blog that I wouldn't have enough content for it to be worth it. I sat down the other night and though of several posts I can write, so it shouldn't take me long to pump out several quality posts. It'll be nice to FINALLY provide my readers with some fun and useful tips and tricks to use in their ColdFusion, Flex and AIR apps.

CFUnited 2010 is supposed to be the last one, unless it's picked up by a new owner at possibly a new location. It was great to see all of my old friends and meet all of my new ones. I attended with a three day package and brought along my lovely wife this time around. While I was attending the various sessions offered throughout the day, she was touring the Leesburg area and enjoying what the Lansdowne Resort had to offer. Everyone in attendance was certainly trying to make the last CFUnited the best they had seen. Despite the painful sunburn I had gotten swimming the weekend before, which many of you got to see first hand, I had a great time. I even got my own shout out from Charlie Arehart during his keynote on the CF Community. Thanks Charlie!

The server outage problem is actually a reoccuring problem. My blog is hosted on my own hardware and recently I've had two machines that seem to be having BIOS issues. At the moment I don't have an available backup supply for my servers and sometimes the AC unit causes the power to drain enough to restart a couple of them. Not the greatest setup, I know. As a result, my email server and database server like to throw a "malprogrammed rom error" at startup. This error goes away with a complete power down and restart, but if they're restarted without a complete power interruption, the error just repeats. It's pretty inconvenient because I actually have to physically be there to fix that problem. I'm in the process of building a new server that will allow me to virtualize a few of my machines, so I won't have this problem any more.

Great to be back and great to have something to write about. I'll see you all again soon!

Well, I finally did it. After several rejections (and to be quite honest mediocre topics on my part), I finally landed my first speaking gig thanks to the folks at NCDevCon. I promise you, if you attend my session, you won't be disappointed. My session is directly after lunch, so if I don't make it a good one I'm sure I'll have a room full of sleepy heads.

My session is titled "Keeping in Sync - Synchronizing client / remote databases in Adobe AIR". One of the very first things I found facinating about AIR, back when it was Apollo, was the built in SQLite database and the ability to push and pull information to and from other databases using Flash Remoting. I was able to utilize this functionality in a prototype AIR application I wrote for the United States Marine Corps. I do mostly ColdFusion at my current job, but I try my best to keep up with advancements in Flex and AIR development on my own time so I can stay fresh. The point of this session is to share with my colleagues how simple it really is to keep a remote database and client database synchronized with some of the great tools built right into AIR and ColdFusion 9. Terry Ryan did a very similar presentation at last year's CFUnited during his keynote. My goal is the same as his, to show how simple it really is to utilize this functionality in your own projects.

Hopefully I'll see you there! If not, I'm sure the sessions will be recorded or the slides will be posted after the conference. Come on down to NCDevCon and join in the fun!