Pluggable Search Components with Search API and FacetAPI

A great opportunity arose a few months back when Tom Nightingale and I were each beginning work on client sites needing advanced search functionality: the Search API module, providing "a framework for easily creating searches on any entity known to Drupal, using any kind of search engine" was becoming more awesome and more stable by the day, and the FacetAPI module had just been announced, but there was no integration between the two.

The attraction to Search API, for the project I was working on at least, arose from some initial uncertainty around whether we'd be using Sphinx or Apache Solr for the backend, and from the fact that we definitely wanted to use a Views front-end for displaying search results.

Collapsible facets

As for faceting, both Tom's project and mine had "collapsible" facets in their designs, something that had already been done on a previous Affinity Bridge client site that used Searchlight on Drupal 6, but it had been implemented as part of the theme. FacetAPI module was offering, among other great features, the possibility of recreating these collapsible facets as a widget that could be re-used across multiple sites.

An issue had already been posted in the Search API issue queue about integrating with FacetAPI, so I jumped in and declared my eagerness to work on it. After several initial iterations of the patch, with tremendous support from Thomas Seidl, author of Search API, and Chris Pliakas, author of FacetAPI, the work was moved into a separate sandbox project where it was easier to collaborate on; and now, a mere 3 months later, the two modules are happily joined in API matrimony.

So what?

So, these are two incredibly awesome modules that you can now use together to create a really great search experience. While the whole is certainly greater than the sum of its parts, it is worthwhile extolling the virtues of each part. I should point out, however, that neither module as a full release yet - Search API being held up mainly by Entity API.

Search API

This is a beautifully architected module that abstracts out all of the concepts involved in the process of site search. Implementations for common backends, e.g. the database and Apache Solr, are provided as separate modules, and the admin UI allows you to add as many search servers and indexes as your site could possibly require (by index here I mean a definition of how your site content is to be indexed - and although each index is associated with a particular server, the latter can be swapped out). To illustrate, your site could have a faceted View page powered by Apache Solr, another one powered by the database, and a block in the sidebar on both of these pages powered by Xapian.

Each backend declares which features it supports, e.g. faceting, autocomplete or spellcheck; and all Search API needs to know is that it supports a given feature, not how it supports it (it is an API module, after all). One of the main things this meant for me, once I'd decided on Solr for a backend, was that contrary to my expectation to have to tweak the solrconfig.xml and schema.xml files (files governing the configuration and index definition, respectively, of the Solr instance) to get the exact behavior I needed, I could actually do everything via the UI - and everything I did via the UI could be exported to my search feature. Perfection.

 

FacetAPI

The FacetAPI module deals with the creation, management and display of facets for search results, independently of the search engine providing these results. Facets can be defined in different realms - a realm being a way of grouping facets together. The only realm that comes with the module is the block realm, allowing you to have each facet in its own block. For each facet you can decide such things as whether it uses OR or AND filtering, the maximum number of facet options to display, whether to display a facet link for documents that don't have a value for that field (the "missing" facet), which searches the facet should appear on, etc.

So now we have converted the "collapsible facets" css and javascript from our old D6 theme to a FacetAPI widget, Facetapi Collapsible, and written another add-on module that provides a new realm which allows you to put all facet links into one block, Facetapi Block, the latter being still very much a work in progress.

Putting it all together

I've set up an install profile with a search feature that you can use to get up and running with Search API and FacetAPI very quickly. These instructions assume you are familar with drush and drush make, and will help you build a brand new Drupal install with everything you need pre-configured, with the exception of the Solr instance. If this is your first time using Solr, don't worry - it's outrageously simple to set up and there are great instructions for doing so here - just do the bit under "Setup Apache Solr", no need to do the rest of it as it's all done for you in the searchtastic feature that comes with the install profile. So, from within a directory that will be the parent directory of your search site, run this little command:
% drush make --working-copy "https://raw.github.com/affinitybridge/d7search/master/stub.make" d7search
This will grab Drupal core and the install profile, which includes another make file which grabs all the required contrib (for more information on building sites this way, read Shawn's recent Build Kit Abridged post). Now cd into the newly created d7search directory and install the site with the following command (making sure to substitute your real db username and password for [username] and [password] respectively):
% drush site-install d7search --site-name="D7 Search" --db-url=mysql://[username]:[password]@localhost/d7search

Assuming you have set up a vhost for this site at d7search.local, login to the new site with the admin/admin creds and go to admin/config/search/search_api and you will see a server and an index set up. If you have solr up and running, then the "Solr Search" server should be able to contact it - click on its name to view its configuration and a message should be displayed at the top telling you that the Solr server could be reached.

Of course, we'll need some content on our site if we want to have anything to index, so go ahead and create some dummy content - from the command line, run:% drush genc --types=story,page,article 100

Then go back to admin/config/search/search_api and click the index's "edit" link and then select "status" from the options popup. On the status page it should tell you that all of your site's content needs to be indexed. Click on the index button as many times as it takes to index your content.

To see everything in action, you can now just go to d7search.local/search and see a page listing your content, with a keyword search block and a facet block in the right sidebar. Play around with keyword searches and facets and you should see that it works nicely.

To see and alter the facet configuration, go back to admin/config/search/search_api and once again click "edit" beside the "Content Search" index and then select "facets" from the options popup. You will see the list of enabled facets. You can enable some more (the list displayed corresponds to the list of fields that is being indexed, which itself can be altered by clicking on the "fields tab" - just remember to re-index again if you do make changes there). Click on the "configure display" link for one of the enabled facets. Here you can change pretty much everything you could possibly want to about the facet's behaviour and display. For example, change the widget used to display the facet - select "links" for just a regular old list of facet links instead of the collapsible list.

With Solr as a backend, you can take advantage of some great search features that it supports, such as spelling suggestions, "more like this" and "OR"-based facets. Let's change the country facet to an OR facet. From the facet settings page, click the "configure display" link for the "Country" facet. Under global settings on the facet config screen, you should see an option to change the filter operator from AND to OR. Go ahead and make this change and click "Save configuration". Now, back on the search page, when you select a country to filter by, you'll still have the other options visible, and clicking on another option will mean seeing results where country is option1 OR country is option2.

There are tons more configuration options both on the Search API side and on the FacetAPI side, but hopefully this has given you a flavour for how great these two modules are, both individually and combined.

Comments

It's great to see such

It's great to see such progress on Drupal Search. Thank you very much for sharing your accomplishments!

Great post!

Great post, Katherine! What you failed to mention is the amazing work you did to integrate the two modules :-). Thanks for the contributions, and the Facetapi Collapsible and Facet API Block modules look awesome.

Thanks, Chris! The Facetapi

Thanks, Chris! The Facetapi Collapsible module is in decent shape, but I need to figure out the right route to take with Facet API block, i.e. whether it should be a block of facet blocks or a separate realm that creates a block of facet links (there's currently a branch for each approach in the sandbox). I just noticed your comment at http://drupal.org/node/593658#comment-5044074 re the Current Search block and would like to help with that - I too feel it needs to be separated out from the facetapi_block stuff in general, because the latter is the only existing example of a realm, so things get a bit confusing. Anyway, it is fun stuff to work on :)

Great Post, But ...

You failed to mention that Search API is D7 only and in dev/beta state atm, which is not very useful.

Also in Search API there is no backend for sphinx which actually integrates with PHP much better then Solr does. There is also Drupal binding for it.

Hi there, Thanks for your

Hi there,
Thanks for your comment. I did actually mention the lack of full releases for both modules just before the section on Search API, though perhaps I should have made it explicit that this meant D7 only.
As regards a Sphinx backend for Search API, it is true that no such implementation currently exists, but wouldn't this just be the perfect opportunity to really use the API? i.e. by writing that implementation - personally, I'd have loved the opportunity to have a go at that if it had turned out we needed Sphinx. I'm not that familiar with it though - can you elaborate on how it integrates better than Solr with PHP? I take it that means you don't need a helper client library like you do with Solr. Or is there more to it?

Error on drush make

When I try to execute the command: drush make --working-copy "https://raw.github.com/affinitybridge/d7search/master/stub.make" d7search

I get:

Project information for drupal retrieved. [ok]
drupal downloaded from http://ftp.drupal.org/files/projects/drupal-7.8.tar.gz. [ok]
Unable to clone d7search from git@github.com:affinitybridge/d7search.git. [error]

Hey there, so, it seems this

Hey there,
so, it seems this has to do with your version of drush make - but don't worry, there's a work around. If you copy the stub.make file locally and just change the git url for the install profile from
git://github.com/affinitybridge/d7search.git
to
http://github.com/affinitybridge/d7search.git
And then run it like this:
% drush make --working-copy stub.make d7search
That should work.

Awesome!

Awesome post Katherine! This is really cool stuff :)

Search Facets not showing

Hi Katherine,

I was able to install the site using your drush make profile and I also installed Apache Solr on my ubuntu 11.10 server. I do get the solr admin screen when I go to my site:8983/solr/admin.

However when i go to the drupal /search screen, I do not see the faceted search blocks, even when I enable the facet fields and enable the blocks.

When I actually perform a search I only get the keyword search block and it always returns 0 items.

I'm not sure what I could have missed. Any advice would be appreciated.

Thanks

Re-installed Solr

I reinstalled Solr and setup a test D7 site with the apachesolr module enabled (Solr and apache are on the same box). That seemed to work fine using Solr to index and search the documents. On the same server, I then re-installed the searchtastic site using the d7search profile. When I try to index the documents, I get an error message in the watchdog log: An error occurred while indexing: '400' Status: Bad Request. Any thoughts on what this could be? Can I use two sites on the same solr instance?

Hey there, are you saying you

Hey there,
are you saying you have one site using apachesolr module and one using Search API with solr, and that you have them both pointing to the same solr instance? You certainly can use the same solr instance but you'll need a multi-core set-up, i.e. the sites will need an index each because the schema is different for sites using the apachesolr module than for sites using Search API + Solr. Are you using the jetty example instance? There are instructions here on running it as a multi-core set-up: http://wiki.apache.org/solr/CoreAdmin#Example. The main thing to remember is that you'll need to copy the necessary schema.xml and solrconfig.xml files into each core directory.
I hope this helps.
Katherine

This is great

I played around with the apache solr search module befor i switched to searach api. its genious. an search api is not beta anymore :)

Repeating Date Search

Hi: I've been looking over your amazing work and want to thank you for a job well done. I was wondering if you have any thoughts or recommendations about searching repeating dates. This seems to be an ongoing pursuit for a lot of people -- I've had a difficult time trying to figure out how to use search api to both display and sort entities on repeating date fields.

I'm using the straight search db module -- not solr -- but I wonder if solr would provide any of this capability. Any thoughts are appreciated. Thanks!

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options