Roxy Deployer and Application Builder

Author: Dave Cassel  |  Category: Software Development

This tutorial has been superseded by a more recent version, which makes the process easier. 

I’ve had a couple people ask me how to use Roxy with MarkLogic‘s Application Builder. The answer has changed recently, so I think it’s time to offer a tutorial. I’m writing this with:

  • MarkLogic 6.0-2.3
  • Roxy 1.4 (see the version.txt file in the project’s base directory)

Motivation

Application Builder makes it easy to create search applications, but copying that application to other servers can be a pain. In my case, I often need to get an application onto a few laptops (each of the developers on a project), a dev server (used for integration testing) and a production server (used to actually show a demo to a prospect).

Assumptions

This tutorial assumes the following:

  • You have installed Ruby (needed for the Roxy deployer)
  • You have downloaded Roxy’s ml script and put it into your path (note: that link points to the latest & greatest on the dev branch)
  • You have installed MarkLogic 6.0-2+
  • You have downloaded MarkLogic Content Pump (needed to export the code from Application Builder to the file system)
  • You are using localhost for your MarkLogic instance
  • All commands are run from your project’s home directory (created by “ml new”) unless otherwise stated

Approach

In this tutorial, I’ll walk through using Roxy and App Builder on a project where that’s the plan from the beginning. (It’s a bit different if you have an existing database and you’ve already run App Builder. I’m leaving that scenario out because I’m intending better tool support.) Here’s the high-level plan:

  1. Create a Roxy project
  2. Set up the configuration
  3. Bootstrap
  4. Run App Builder
  5. Capture the code
  6. Cleanup

Create a Roxy Project

Roxy lets you create a new project using the ml new command:

$ ml new ab-app

Using the command this way will create a Roxy MVC application, intended for projects built mostly from XQuery. As of Roxy 1.4 (currently on the dev branch), you can also choose to do a rest application (like what Application Builder produces — using the REST API with no MVC XQuery behind it) or a hybrid application (using the REST API and Roxy’s MVC structure). For this tutorial, I’ll do a REST application, although in practice I’d be more likely to do a hybrid one.

# from ~/git/, or wherever you like your projects to live:
$ ml new my-app --branch=dev --app-type=rest
$ cd my-app

This creates a Roxy application with the name “my-app”. That name will be used in naming app servers, databases, and other things set up by Roxy for this application.

The REST application type provides no source code, just the REST API set up, on the assumption that you’ll provide the source code. After running the command above, you’ll have a deploy directory, containing the configuration of your databases, forests, app servers, roles and users; a rest-api directory, where you can put the options that will control your REST API searches; and a src directory, which for a REST application will be empty.

Set up the Configuration

Our next step is to set up the configuration that we need. In particular, you’ll want to add range indexes and make other changes to the database settings. You can do this by editing deploy/ml-config.xml (there are comments in the file providing examples) or by using the index interview feature (“$ ml index” will ask you a set of questions and build element or attribute range indexes based on the answers). Having the range indexes in place will give Application Builder something to work with.

Other than the indexes, you might need to change the ports or password that Roxy will use. In deploy/build.properties, you’ll see the default values for app-port, xcc-port, user and password. Change them as needed to avoid conflict with existing applications and to match your admin username and password. (If you don’t want the password in your properties file, leave it blank (“password=”); Roxy will prompt you.

Bootstrap

Time put that configuration to work.

$ ml local bootstrap

After running bootstrap, you should have an HTTP app server, and XCC app server, a content database and a modules database. If you have some sample data to work with, create a {project-home}/data directory, put the data there, and run

$ ml local deploy content

We don’t have any code yet, so there’s no need to deploy modules.

At this stage, we have something worth preserving — the configuration. That makes it a good time to set up a local git repository.

$ git init
$ git add *
$ git commit -m "Initial commit"

Run Application Builder

We have a database with indexes and some content (assuming you loaded some sample documents). On to Application Builder. This tutorial doesn’t cover how to use Application Builder, but here are the relevant parts for our purposes:

  1. Point your browser to http://localhost:8000 and click Application Builder at the top. 
  2. Click the New Application button
  3. Pick an application name different from the name you gave Roxy (“my-app” above; use “ab-my-app” or something)
  4. For the database, pick the content database built by Roxy (“my-app-content”)
  5. Go through the wizard
  6. At the Deploy stage, choose a port that does not conflict with the one Roxy is using

You now have two applications: a Roxy app with no source code and an Application Builder app. Both point to the same content database.

Capture the Code

Next step, we’ll capture the code. We’ll use the MarkLogic ContentPump (mlcp) for this. mlcp is a great tool for this job, apart from one weakness: the database you get data from is based on the content database for the XDBC app server that you use. A more flexible solution would let you specify the content database, as we can do using the connection string with recordloader, xqsync, and corb. We’ll work around that.

We’re going to use mlcp to export the contents of the modules database that Application Builder created and put them under src. To do this, we need an XDBC app server. Happily, Roxy created one for us; however, it’s pointing to my-app-content and we need it to point to my-app-modules.

  • Use the admin UI (http://localhost:8001) to change my-app-xcc’s database setting to point to ab-my-app-modules (adjust names based on what you picked for the Roxy app and the App Builder app)
  • mlcp won’t write to a directory that already exists. There’s no option to override, so
$ rmdir src
    • Note that if we were doing a hybrid application, there would be other source code in the src directory, and deleting src would be a bad idea. In that case, export Application Builder’s modules database to some other location, then copy it all to the src directory.
  • Now run the export:
$ ~/software/marklogic-contentpump-1.0.2/bin/mlcp.sh export \
-host localhost -port 8061 -username admin -password admin -output_file_path src
    • Your path to mlcp is likely different. Your port, username, and password are likely different. Adjust accordingly.
  • Having done the export, use the admin UI (http://localhost:8001) to change the my-app-xcc’s database setting to point back to my-app-content
  • Now we’ll move the rest-api options to a special directory so that the deployer can make sure they end up in the right place.
    • Note that we could skip this step and leave them where they are. Everything would be fine as long as you don’t change the name of the application or the group that the appservers are part of. If either changes, the URIs of the rest options need to change as well. To make sure they end up where they need to be, we’ll let the Roxy Deployer figure out their URIs based on the app’s configuration.
    • If you do want to skip this step and leave the rest options where they are, remove {project-home}/rest-api/options/all.xml. This is a sample set of options created with the Roxy project and will overwrite the ones under src.
# Remove the sample REST API options created by Roxy:
$ rm -rf rest-api
# Move App Builder's REST API options to the rest-api directory
$ mv src/Default/ab-my-app/rest-api .
# Get rid of the now-empty directory that used to hold the options
$ rm -rf src/Default

This is another good time to capture what we have.

$ git add rest-api src
$ git commit -m "Added App Builder code and options"

Cleanup

This step is optional. You might want to re-run App Builder a couple time to tweak what you have, and the repeat the process of grabbing the code. If you decide you want another index, make the change in deploy/ml-config.xml and then bootstrap again — doing it this way (instead of through the Admin UI) keeps your configuration up to date.

When you are done with the App Builder version, you can remove that project.

  • On http://localhost:8000/appbuilder, you can delete the App Builder project
  • Using the Admin UI on http://localhost:8001, you can delete the HTTP app server, the App Builder-generated modules database, and the associated forest.

Deploying to Other Environments

We have captured the App Builder version of the application. Now suppose we want to set up a copy on some other host “devbox” as our dev environment.

  1. Create a file called deploy/dev.properties
  2. If the ports you used locally are not available on devbox, copy the app-port and xcc-port properties from deploy/build.properties to deploy/dev.properties
  3. If the admin password is different on devbox, copy the password property from build.properties to dev.properties (or leave it blank: “password=”)
  4. Set the dev-server property to the name of the server — you’ll probably need a fully qualified domain name

At that point, you should be set.

$ ml dev boostrap
$ ml dev deploy modules
$ ml dev deploy content

Happy coding!

Tags: , , ,

One Response to “Roxy Deployer and Application Builder”

  1. David Cassel » Blog Archive » Capturing an App Builder application with Roxy Says:

    […] post is an update to my earlier tutorial on how to use Roxy with MarkLogic‘s Application Builder. I finally wrapped up a feature that […]

Leave a Reply