David Cassel

Over the last week, I conducted a survey of how people use the Roxy framework. I got 34 responses. Let’s see what people said. Note that multiple answers were allowed to the first four questions (Roxy Components, MarkLogic Versions, Roxy Application Types, Architectures).

Amazon Gift Card

I’ll start with the results of the last question: “ To be eligible for the $10 Amazon card drawing, enter your email address (no spam, I promise). Winner will be notified by email and announced on http://davidcassel.net.”

22 participants provided their email addresses for this question. I inserted them into an empty MarkLogic database, like so:

for $e in $emails 
return
  xdmp:document-insert(
    '/' || xdmp:random() || ".xml",
    <email>{$e}</email>
  )

and then ran a randomized search:

cts:search(
  fn:doc(),
  cts:element-query(xs:QName('email'), cts:and-query(())),
  'score-random'
)[1]

I have sent an email to the winner to confirm that the email address is valid.

On to the survey results.

Roxy Components

What parts of Roxy do you use?

I anticipated that everyone would select the Deployer and a few would pick Unit Testing. I was very curious to see the mix between XQuery MVC and REST.

1. Deployer (88.2%, 30)
2. XQuery MVC code structure (73.5%, 25)
3. REST API scaffolding & configuration (50%, 17)
4. Unit testing (35.3%, 12)

I was very surprised that Deployer was not 100% — I’m actually not sure how someone uses Roxy without the deployer. (If you do, I’d be very interested to hear from you.) More people are using Unit Testing than I thought, which I’m glad to see.

The mix between MVC and REST is interesting. Given that Roxy originally only supported MVC, I’m not surprised to see that option get a high percentage, but it looks like the REST support is getting some good use as well.

Of the 12 who reported using the Unit Testing feature, 5 are ML Professional Services, 5 are customers, 1 is a MarkLogic Other, and 1 is a partner.

MarkLogic Versions

With what versions are MarkLogic do you have active or planned Roxy projects?

1. MarkLogic 4 (5.9%, 2)
2. MarkLogic 5 (41.2%, 14)
3. MarkLogic 6 (82.4%, 28)
4. MarkLogic 7 (52.9%, 18)

At this time, the most recently released version is MarkLogic 6, so there’s no surprise that it got the highest number of results. MarkLogic 7 has not yet been released, but is available to MarkLogic employees and Early Access program participants. I’m not surprised to see a number of MarkLogic 5 deployments, but I didn’t realize Roxy is being used with MarkLogic 4 still — very interesting. We have made an effort to keep Roxy backwards compatible. Some new features, such as anything to do with the REST API, require more recent versions.

Of the two respondents who reported using ML4, one is a customer, one is a partner. Among ML5 users, 5 are customers, 4 are Professional Services, 3 are ML Other, and 2 are partners.

Roxy Application Types

Which of the Roxy application types do you use for new projects? (–app-type parameter on new/init)

1. MVC (XQuery applications using the Model View Controller pattern) (44.1%, 15)
   
2. REST (Applications based on the MarkLogic REST API; no XQuery code provided) (35.3%, 12)
3. Hybrid (MVC + REST) (58.8%, 20)
4. I don’t use Roxy to manage my source code (17.6%, 6)

I added the last option after one of my colleagues reviewed my survey questions, and I’m glad I did. The Deployer is a useful tool, even if you choose to blow away the src directory and use your own structure.

Hybrid won this question. My own use of Hybrid has been a gateway: I started with MVC, wanted to work with REST, but wasn’t yet ready to give up what I was used to.

Architectures

In what kind of architectures do you use Roxy?

1. Single-tier XQuery-based UI (HTML pages and back-end both built with XQuery) (50.0%, 17)
   
2. Single-tier JavaScript-based UI (UI primarily built with HTML/JS hosted by MarkLogic; back-end implemented w/MarkLogic) (64.7%, 22)
3. XML Services (implement services using XML; UI hosted elsewhere) (55.9%, 19)
4. JSON Services (implement services using JSON; UI hosted elsewhere) (29.4%, 10)
5. Other (11.8%, 4)

Of the four “Other” answers, I believe two of them involve hosting the GUI. Digging a little deeper into the responses, 29 of the survey participants reported one of the architecture choices that involves MarkLogic hosting the UI.

Participant Roles

Are you currently a:

1. MarkLogic Professional Services employee (29.4%, 10)
   
2. Other MarkLogic employee (23.5%, 8)
3. MarkLogic partner (14.7%, 5)
4. MarkLogic customer (32.4%, 11)

That’s a nice mix. I don’t actually know how many people are using Roxy overall, so it’s hard to say what percentage of users responded to the survey. But I’m happy to a mix of MarkLogic employees, partners, and customers working with Roxy.

Wrap Up

In retrospect, a really useful question would be “Have you used Roxy on a production application?” Maybe I’ll do another survey sometime in the future.

If you think of another way to look at the data that would be interesting, leave a comment.

Thank you to all the participants, this was very helpful for me!

For those who didn’t see my email announcement or Tweet, I’m conducting a 5-question survey of how people use Roxy with MarkLogic. Your responses will help me focus future development work on the types of features that will help the most people. So far I have 26 responses and have already seen some interesting results. I plan to post results of the survey Friday — stay tuned!

I just sent this out to the Roxy mailing list.

Hello all,

I’m happy to announce Roxy 1.4 has just moved up to the master branch. That means lots of new features when you create a new project, without having to specify the dev branch.

$ ml new my-app --server-version=6 --app-type=hybrid

Roxy 1.4 added several features related to the REST API, and introduced the rest and hybrid application types. The Roxy deployer now supports additional commands: capture and upgrade, as well having additional capabilities while bootstrapping your project. The documentation has improved, including the beginning of a series of video tutorials about how to use Roxy.

New work will continue on the dev branch as Roxy 1.5. Submit a ticket for features or tutorials you’d like to see. Commenting on existing tickets lets us know what’s important.

Here’s the full change log for version 1.4 (closed tickets):

New Features

• #63 Use –app-type parameter with ml new to create mvc, rest, or hybrid apps
• #82 Use Corb with the deployer
• #88 Deploy REST API extensions
• #90 Allowed setting rewriter and error handler to blank
• #92 Scaffolding for REST API extensions
• #94 Create ODBC app servers
• #97 Added a test-authentication-method property, allowing for different methods between test and main
• #101 Deploy schemas command
• #102 Hide password characters as they are typed
• #103 Bootstrap an HTTP server with SSL
• #107 Deploy REST API transformations
• #108 Create skeleton REST API transformations
• #112 Added ml capture command
• #113 Override response types
• #117 Support new ML7 database settings
• #120 Require server-version parameter for ml new/init
• #121 Added “ml upgrade” command

Resolved Issues

• #93 Fixed the deployer so it works with ML7 EA1.
• #97 Test app server now gets same config as the main one, except where differentiated
• #104 Deployer was ignoring the forest-data-dir property
• #110 Updated bootstrap procedure to work with new ML7 endpoint
• #118 Apply field settings
• #119 Make Roxy more forgiving about import error
• #123 Accept a broader range of affirmative responses when upgrading
• #124 Fixed corb integration bug
• #126 Don’t overwrite application/custom when running “ml capture”

On a few different people’s recommendations, I’ve been looking at AngularJS lately. Something I ran into early was the expectation that you use HTML5, which is generally not valid XML (meta and link elements aren’t closed, and AngularJS uses some valueless directives, like <body ng-app>). MarkLogic also gets confused by the {{ }} binding markers in a file it thinks might be XQuery. That means we need to store the HTML as text in MarkLogic, which is fine if you’re using straight-up HTML, but what about with XQuery-based views & layouts?

Looking into using Angular with Roxy‘s MVC application structure, I find myself leaning toward using the MVC approach to build services and defer nearly the entire UI to JavaScript on the front-end. So how do we do that in a Roxy MVC project?

Here are the steps I came up with.

1. Create src/public/html/
2. Create one or more .html files in that directory
3. Instruct the Roxy deployer to load HTML files as text
4. Add rules to the rewriter so that the HTML files are accessible

The first two steps are pretty self explanatory. To tell Roxy to load HTML files as text, we copy the load-html-as-xml property from deploy/default.properties to deploy/build.properties:

load-html-as-xml=false

Now let work on the rewriter. We’re going to add two rules. One will take any request that ends with .html and direct it to the public/html directory:

<request uri="^/(.*html)" endpoint="/public/html/$1"/>

Note that this has the effect of masking requests to MVC endpoints using the html format. With this approach, we’re committed! (Note on an alternative approach below.)

We should also set a default home page. Normally, Roxy MVC defaults to /<default-controller>/<default-function>.<default-format>, for instance, /appbuilder/main.html. We’re going to set it to an html page:

<request uri="^/$" endpoint="/public/html/index.html"/>

… assuming that we have created an index.html. With these changes, here’s how $c:ROXY-ROUTES in src/app/config/config.xqy looks:

declare variable $c:ROXY-ROUTES :=
  <routes xmlns="http://marklogic.com/appservices/rest">
    <request uri="^/$" endpoint="/public/html/index.html"/>
    <request uri="^/(.*html)" endpoint="/public/html/$1"/>
    {
      $def:ROXY-ROUTES/rest:request
    }
  </routes>;

Another helpful step we can take is to choose a default format other than HTML. We can do so in src/app/config/config.xqy, changing $c:ROXY-OPTIONS:

declare variable $c:ROXY-OPTIONS :=
  <options>
    <default-format>json</default-format>
    <layouts>
      <layout format="html">three-column</layout>
    </layouts>
  </options>;

This means that when a client hits an endpoint with no format specified, like /users/profile, it will be assume a json response format instead of html. Clients can explicitly specify this as /users/profile.json, as usual.

Alternative Approach

As mentioned above, this approach masks MVC endpoint requests with the HTML format. For instance, if we ask for /search/main.html, the rewrite rules above will redirect to /public/html/search/main.html, rather than the search controller’s main function. I’m okay with that, because I want to use Angular to build my UI on top of an HTML template. Here are two other approaches you can use, if you want to keep the ability to hit HTML endpoints.

Use the default format

The routing path above rewrites any path that ends in html to the static html directory. However, if we leave the default-format option with the original html, then we can still use a traditional view/template by using a path like /user/add-form and get back an HTML page.

More specific route path

The routing path above is pretty course: if it ends in html, it’s rewritten to /public/html/. We could also be more specific, like requiring paths to HTML to have a longer path, like /html/user-form.html. Then the path in $c:ROXY-ROUTES could be uri=”^/html/(.*)$” endpoint=”/public/html/$1″. Personally, I don’t like the URLs you get from this approach.

AngularJS and the REST API

If you’re doing a Roxy application with app-type=rest, then the HTML part of this is easy: that kind of application expects HTML files. You’ll still need to tell Roxy not to load them as XML. From that point, the trick is to get Angular to work with the REST API. I haven’t done that yet, but if I do, you’ll see it here.

This post is an update to my earlier tutorial on how to use Roxy with MarkLogic‘s Application Builder. I finally wrapped up a feature that automates the process, so the procedure has changed. I’m working with:

• MarkLogic 6.0-3.1 (6.0-2+ required)
• Roxy 1.4 (see the version.txt file in the project’s base directory)

Motivation

Application Builder makes it easy to create search applications, but copying that application to other servers can be a pain. In my case, I often need to get an application onto a few laptops (each of the developers on a project), a dev server (used for integration testing) and a production server (used to actually show a demo to a prospect). Capturing the application’s REST API configuration and source code on the file system allow us to get it under version control, making it much easier to work in a team environment.

Assumptions

This tutorial assumes the following:

• You have installed Ruby (needed for the Roxy deployer)
• You have downloaded Roxy’s ml script and put it into your path (note: that link points to the latest & greatest on the dev branch)
• You have installed MarkLogic 6.0-2+
• You are using localhost for your MarkLogic instance (though you can capture from remote servers as well)
• All commands are run from your project’s home directory (created by “ml new”) unless otherwise stated

Approach

In this tutorial, I’ll walk through using Roxy and App Builder on a project where that’s the plan from the beginning. (It’s a bit different if you have an existing database and you’ve already run App Builder. I’m leaving that scenario out because I’m intending better tool support.) Here’s the high-level plan:

1. Create a Roxy project
2. Set up the configuration
3. Bootstrap
4. Run App Builder
5. Capture the code
6. Cleanup

Create a Roxy Project

Roxy lets you create a new project using the ml new command:

$ ml new app-name

Using the command this way will create a Roxy MVC application, intended for projects built mostly from XQuery. As of Roxy 1.4 (currently on the dev branch), you can also choose to do a rest application (like what Application Builder produces — using the REST API with no MVC XQuery behind it) or a hybrid application (using the REST API and Roxy’s MVC structure). This approach to capturing App Builder code requires having a REST application (in the future, I could set it up to work with hybrid apps if people request that).

# from ~/git/, or wherever you like your projects to live:
$ ml new my-app --branch=dev --app-type=rest
$ cd my-app

This creates a Roxy application with the name “my-app”. That name will be used in naming app servers, databases, and other things set up by Roxy for this application. That’s also the name of the directory where the project lives (~/git/my-app, in my case).

The REST application type provides no source code, just the REST API set up, on the assumption that you’ll provide the source code. After running the command above, you’ll have a deploy directory, containing the configuration of your databases, forests, app servers, roles and users; a rest-api directory, where you can put the options that will control your REST API searches; and a src directory, which for a REST application will be empty.

Set up the Configuration

Our next step is to set up the configuration that we need. In particular, you’ll want to add range indexes and make other changes to the database settings. You can do this by editing deploy/ml-config.xml (there are comments in the file providing examples) or by using the index interview feature (“$ ml index” will ask you a set of questions and build element or attribute range indexes based on the answers). Having the range indexes in place will give Application Builder something to work with.

Other than the indexes, you might need to change the ports or password that Roxy will use. In deploy/build.properties, you’ll see the default values for app-port, xcc-port, user and password. Change them as needed to avoid conflict with existing applications and to match your admin username and password. (If you don’t want the password in your properties file, leave it blank (“password=”); Roxy will prompt you.

Bootstrap

Time put that configuration to work.

$ ml local bootstrap

After running bootstrap, you should have an HTTP app server, and XCC app server, a content database, a modules database, and an app-specific role and user. If you have some sample data to work with, create a {project-home}/data directory, put the data there, and run

$ ml local deploy content

We don’t have any code yet, so there’s no need to deploy modules.

At this stage, we have something worth preserving — the configuration. That makes it a good time to set up a local git repository. This is a good idea even if you’re just working by yourself on a small project.

$ git init
$ git add *
$ git commit -m "Initial commit"

Run Application Builder

We have a database with indexes and some content (assuming you loaded some sample documents). On to Application Builder. This tutorial doesn’t cover how to use Application Builder, but here are the relevant parts for our purposes:

1. Point your browser to http://localhost:8000 and click Application Builder at the top.
2. Click the New Application button
3. Pick an application name different from the name you gave Roxy (“my-app” above; use “ab-my-app” or something)
4. For the database, pick the content database built by Roxy (“my-app-content”)
5. Go through the wizard
6. At the Deploy stage, choose a port that does not conflict with the one Roxy is using

You now have two applications: a Roxy app with no source code and an Application Builder app. Both point to the same content database.

Capture the Code

App Build creates a MarkLogic REST API-based application consisting of HTML, JavaScript, CSS, images, and some configuration options for the REST API. The ml capture command retrieves these files from the modules database created by App Builder and puts them on the file system, in your project, in the directories where things belong. Here’s how to use it:

$ ml local capture --modules-db=ap-my-app-modules

Let’s break that down.

The local part specifies that the App Builder modules database is available on the local environment server. If you ran App Builder somewhere else, you can use a different environment to get there. For instance, if my dev environment points to van-dev2 and I’ve run App Builder there, I can use dev in this command instead of local.

The –modules-db parameter specifies the name of the modules database created by App Builder.

The command will create a temporary directory on your system (for instance, somewhere under /tmp/ on Linux), download the files, then copy them into the appropriate places in your project. For the source code, this would typically be under your project’s src/ directory, but if you’ve changed the xquery.dir property to point somewhere else, that’s where the source code will go. Likewise, the REST API configuration will be copied to the directory pointed to by your rest-options.dir directory, typically rest-api/config/.

Note that App Builder typically generates one transform (appidentitytransform) and one extension (extsuggest). When you create transforms and extensions in a Roxy application, they would typically go under rest-api/transform or rest-api/ext (see Working With the REST API for more information). The capture command currently leaves the App Builder-generated transform and extension under src. That’s okay; deploy modules knows what to do with them. Move along.

We can test that the capture was successful by deploying the code to our Roxy application server.

$ ml local deploy modules

If all went well, you should now be able to point your browser to your Roxy app-port and see the captured version of the app. It should be identical to the App Builder version.

This is another good time to capture what we have.

$ git add rest-api src
$ git commit -m "Added App Builder code and options"

Cleanup

This step is optional. You might want to re-run App Builder a couple time to tweak what you have, and the repeat the process of grabbing the code. If you decide you want another index, make the change in deploy/ml-config.xml and then bootstrap again — doing it this way (instead of through the Admin UI) keeps your configuration up to date.

When you are done with the App Builder version, you can remove that project.

• On http://localhost:8000/appbuilder, you can delete the App Builder project
• Using the Admin UI on http://localhost:8001, you can delete the HTTP app server, the App Builder-generated modules database, and the associated forest.

Deploying to Other Environments

We have captured the App Builder version of the application. Now suppose we want to set up a copy on some other host “devbox” as our dev environment.

1. Create a file called deploy/dev.properties
2. If the ports you used locally are not available on devbox, copy the app-port and xcc-port properties from deploy/build.properties to deploy/dev.properties
3. If the admin password is different on devbox, copy the password property from build.properties to dev.properties (or leave it blank: “password=”)
4. Set the dev-server property to the name of the server — you’ll probably need a fully qualified domain name

At that point, you should be set.

$ ml dev boostrap
$ ml dev deploy modules
$ ml dev deploy content

Happy coding!

This tutorial has been superseded by a more recent version, which makes the process easier. 

I’ve had a couple people ask me how to use Roxy with MarkLogic‘s Application Builder. The answer has changed recently, so I think it’s time to offer a tutorial. I’m writing this with:

• MarkLogic 6.0-2.3
• Roxy 1.4 (see the version.txt file in the project’s base directory)

Motivation

Application Builder makes it easy to create search applications, but copying that application to other servers can be a pain. In my case, I often need to get an application onto a few laptops (each of the developers on a project), a dev server (used for integration testing) and a production server (used to actually show a demo to a prospect).

Assumptions

This tutorial assumes the following:

• You have installed Ruby (needed for the Roxy deployer)
• You have downloaded Roxy’s ml script and put it into your path (note: that link points to the latest & greatest on the dev branch)
• You have installed MarkLogic 6.0-2+
• You have downloaded MarkLogic Content Pump (needed to export the code from Application Builder to the file system)
• You are using localhost for your MarkLogic instance
• All commands are run from your project’s home directory (created by “ml new”) unless otherwise stated

Approach

In this tutorial, I’ll walk through using Roxy and App Builder on a project where that’s the plan from the beginning. (It’s a bit different if you have an existing database and you’ve already run App Builder. I’m leaving that scenario out because I’m intending better tool support.) Here’s the high-level plan:

1. Create a Roxy project
2. Set up the configuration
3. Bootstrap
4. Run App Builder
5. Capture the code
6. Cleanup

Create a Roxy Project

Roxy lets you create a new project using the ml new command:

$ ml new ab-app

Using the command this way will create a Roxy MVC application, intended for projects built mostly from XQuery. As of Roxy 1.4 (currently on the dev branch), you can also choose to do a rest application (like what Application Builder produces — using the REST API with no MVC XQuery behind it) or a hybrid application (using the REST API and Roxy’s MVC structure). For this tutorial, I’ll do a REST application, although in practice I’d be more likely to do a hybrid one.

# from ~/git/, or wherever you like your projects to live:
$ ml new my-app --branch=dev --app-type=rest
$ cd my-app

This creates a Roxy application with the name “my-app”. That name will be used in naming app servers, databases, and other things set up by Roxy for this application.

The REST application type provides no source code, just the REST API set up, on the assumption that you’ll provide the source code. After running the command above, you’ll have a deploy directory, containing the configuration of your databases, forests, app servers, roles and users; a rest-api directory, where you can put the options that will control your REST API searches; and a src directory, which for a REST application will be empty.

Set up the Configuration

Our next step is to set up the configuration that we need. In particular, you’ll want to add range indexes and make other changes to the database settings. You can do this by editing deploy/ml-config.xml (there are comments in the file providing examples) or by using the index interview feature (“$ ml index” will ask you a set of questions and build element or attribute range indexes based on the answers). Having the range indexes in place will give Application Builder something to work with.

Other than the indexes, you might need to change the ports or password that Roxy will use. In deploy/build.properties, you’ll see the default values for app-port, xcc-port, user and password. Change them as needed to avoid conflict with existing applications and to match your admin username and password. (If you don’t want the password in your properties file, leave it blank (“password=”); Roxy will prompt you.

Bootstrap

Time put that configuration to work.

$ ml local bootstrap

After running bootstrap, you should have an HTTP app server, and XCC app server, a content database and a modules database. If you have some sample data to work with, create a {project-home}/data directory, put the data there, and run

$ ml local deploy content

We don’t have any code yet, so there’s no need to deploy modules.

At this stage, we have something worth preserving — the configuration. That makes it a good time to set up a local git repository.

$ git init
$ git add *
$ git commit -m "Initial commit"

Run Application Builder

We have a database with indexes and some content (assuming you loaded some sample documents). On to Application Builder. This tutorial doesn’t cover how to use Application Builder, but here are the relevant parts for our purposes:

1. Point your browser to http://localhost:8000 and click Application Builder at the top. 
2. Click the New Application button
3. Pick an application name different from the name you gave Roxy (“my-app” above; use “ab-my-app” or something)
4. For the database, pick the content database built by Roxy (“my-app-content”)
5. Go through the wizard
6. At the Deploy stage, choose a port that does not conflict with the one Roxy is using

You now have two applications: a Roxy app with no source code and an Application Builder app. Both point to the same content database.

Capture the Code

Next step, we’ll capture the code. We’ll use the MarkLogic ContentPump (mlcp) for this. mlcp is a great tool for this job, apart from one weakness: the database you get data from is based on the content database for the XDBC app server that you use. A more flexible solution would let you specify the content database, as we can do using the connection string with recordloader, xqsync, and corb. We’ll work around that.

We’re going to use mlcp to export the contents of the modules database that Application Builder created and put them under src. To do this, we need an XDBC app server. Happily, Roxy created one for us; however, it’s pointing to my-app-content and we need it to point to my-app-modules.

• Use the admin UI (http://localhost:8001) to change my-app-xcc’s database setting to point to ab-my-app-modules (adjust names based on what you picked for the Roxy app and the App Builder app)
• mlcp won’t write to a directory that already exists. There’s no option to override, so

$ rmdir src

• Note that if we were doing a hybrid application, there would be other source code in the src directory, and deleting src would be a bad idea. In that case, export Application Builder’s modules database to some other location, then copy it all to the src directory.
• Now run the export:

$ ~/software/marklogic-contentpump-1.0.2/bin/mlcp.sh export \
-host localhost -port 8061 -username admin -password admin -output_file_path src

• Your path to mlcp is likely different. Your port, username, and password are likely different. Adjust accordingly.
• Having done the export, use the admin UI (http://localhost:8001) to change the my-app-xcc’s database setting to point back to my-app-content
• Now we’ll move the rest-api options to a special directory so that the deployer can make sure they end up in the right place.
  • Note that we could skip this step and leave them where they are. Everything would be fine as long as you don’t change the name of the application or the group that the appservers are part of. If either changes, the URIs of the rest options need to change as well. To make sure they end up where they need to be, we’ll let the Roxy Deployer figure out their URIs based on the app’s configuration.
  • If you do want to skip this step and leave the rest options where they are, remove {project-home}/rest-api/options/all.xml. This is a sample set of options created with the Roxy project and will overwrite the ones under src.

# Remove the sample REST API options created by Roxy:
$ rm -rf rest-api
# Move App Builder's REST API options to the rest-api directory
$ mv src/Default/ab-my-app/rest-api .
# Get rid of the now-empty directory that used to hold the options
$ rm -rf src/Default

This is another good time to capture what we have.

$ git add rest-api src
$ git commit -m "Added App Builder code and options"

Cleanup

This step is optional. You might want to re-run App Builder a couple time to tweak what you have, and the repeat the process of grabbing the code. If you decide you want another index, make the change in deploy/ml-config.xml and then bootstrap again — doing it this way (instead of through the Admin UI) keeps your configuration up to date.

When you are done with the App Builder version, you can remove that project.

• On http://localhost:8000/appbuilder, you can delete the App Builder project
• Using the Admin UI on http://localhost:8001, you can delete the HTTP app server, the App Builder-generated modules database, and the associated forest.

Deploying to Other Environments

We have captured the App Builder version of the application. Now suppose we want to set up a copy on some other host “devbox” as our dev environment.

1. Create a file called deploy/dev.properties
2. If the ports you used locally are not available on devbox, copy the app-port and xcc-port properties from deploy/build.properties to deploy/dev.properties
3. If the admin password is different on devbox, copy the password property from build.properties to dev.properties (or leave it blank: “password=”)
4. Set the dev-server property to the name of the server — you’ll probably need a fully qualified domain name

At that point, you should be set.

$ ml dev boostrap
$ ml dev deploy modules
$ ml dev deploy content

Happy coding!

I’ve gotten very accustomed to using the Roxy with my MarkLogic projects. However, with the new App Builder available with MarkLogic 6, some of my projects are based on JavaScript and the REST API. I still want to use the Roxy deployer, however. The simplicity of deploying configuration and code with a few simple commands isn’t something I’m willing to give up easily. Roxy doesn’t handle REST API or App Builder applications out of the box just yet (yet though I’ve filed an RFE and hope to have some time to work on it). But with a few modifications, I can still use the Roxy deployer for these projects. Here’s how.

Set the Rewriter

MarkLogic lets you specify the module that will handle URL rewriting, and Roxy provides a property to set this. In deploy/build.properties, add this line:

url-rewriter=/MarkLogic/rest-api/rewriter.xqy

The default Roxy rewriter doesn’t include REST API rewrites, and isn’t needed for an App Builder application anyway.

Set the error handler

Likewise, you can set the module that MarkLogic will call whenever there’s an uncaught exception, so that you can provide a nice error screen to your users. You can tell Roxy to use App Builder’s error page by adding this to deploy/build.properties:

error-handler=/MarkLogic/rest-api/error-handler.xqy

Global Rewrites

A new setting in MarkLogic 6 controls the scope of modules that can be used in rewriting. Global scope allows for MarkLogic-provided modules to be used for this purpose. We’ll need that, so in deploy/ml-config.xml, add this line to the http app server definition (/configuration/group:http-servers/group:http-server/group:rewrite-resolves-globally):

<rewrite-resolves-globally>true</rewrite-resolves-globally>

Add a collection

In the modules database, the files with the URI matching /marklogic.rest* need to be added to a collection. There are two steps to do this. First, find the modules database definition in deploy/ml-config.xml (look for “<database-name>@ml.modules-db</database-name>”), find the uri-lexicon setting and set it to true. (This is something I often do anyway.)

Update: I just checked in a change to the Roxy dev branch that handles this step. Add an “app-type=mvc” property to deploy/build.properties and the “ml deploy modules” command will take care of setting the collection shown below. 

Next, open deploy/app_specific.rb and add this:

def apply_modules_collections()
  r = execute_query %Q{
    xquery version "1.0-ml";

    for $uri in cts:uri-match("/marklogic.rest.*")
    return xdmp:document-set-collections($uri, "http://marklogic.com/extension/plugin")
  },
  { :db_name => @properties["ml.app-modules-db"] }
end

Get rid of the MVC code

This is an App Builder project, so you won’t be using Roxy’s default MVC code. We’ll ditch all of it and replace it with what we get from App Builder.

# from the project directory: (yes, this will delete the whole src directory.
# Make sure you're doing that on purpose.)
$ rm -rf src/*

Get the App Builder code

I’m assuming you’ve already run through the App Builder wizard and have a project. You can then use mlcp or WebDAV to get the generated source code and copy it to your src directory.

Document

In my group, we’re starting to make this common practice: replace the contents of the provided README.markdown with the set of steps needed to deploy the project. Here they are:

$ ml local bootstrap
$ ml local deploy modules
$ ml local apply_modules_collections
# and if you have some sample data:
$ ml local deploy content

Coming Soon(?)

My intention is that at some point in the future, you’ll be able to specify what kind of project you want when you create one. Right now, you make a new Roxy project like this:

$ ml new {new-app-name}

I picture something choosing between options like this:

$ ml new {new-app-name} --rest
$ ml new {new-app-name} --mvc

Most likely MVC will remain the default, but the –rest option would take care of the steps above — including applying the collection to the marklogic.rest* files so that the extra step isn’t necessary. Feel free to comment here or on the GitHub ticket to encourage us to get around to this feature.

Storing Documents, Semantics, and Values in an Enterprise NoSQL Database

In recent years, NoSQL databases have called attention to some of the limitations of relational databases. An RDBMS is great if you know what your data will look like, it fits well into rows and columns, and isn’t likely to change. NoSQL databases have given us new approaches for storing, searching, and retrieving data, with new families like document store, graph databases, key-value stores, and triple stores. Each of these is optimized for different types of data.

Storing data differently changes how applications are built on top of the database. In this session, we’ll investigate the implications of using document stores and triple stores on application development, then see what happens when these two types can be combined. Shifting from the relational model to using documents and triples impacts applications throughout the stack, not just in the database tier. This allows faster application development, the intelligence of semantics, and easier integration of multiple data sources, without having to compromise the features you expect from your database.

Presented at the O’Reilly Software Architecture Conference, 2015 Part 1 Part 2

Roxy Tutorials

Roxy is a 3-part MarkLogic project: a Deployer, to set up databases, application servers, forests, and security, along with deploying source code; a Unit Test framework; and an MVC structure for XQuery applications. Over the years, the Deployer has emerged as the most important component of the three.

Roxy Tutorial videos

My first effort with the DemoCat application had a pretty simple architecture: node hosted static resources and passed REST requests straight through to MarkLogic. This approach is more in keeping with the three-tier architecture that MarkLogic’s customers typically use, but my middle tier was doing almost nothing. I’ve made some improvements, especially related to user management. I should point out that I’m new to node, so if this looks off-base, let me know, but this looks like moving the right direction.

Current Approach

In my first effort, node.js hosted the HTML/JS/CSS; the UI was built with AngularJS; and the node layer proxied requests to MarkLogic’s REST API straight through without looking at them at all.

What’s Wrong with Pass-through

The problem with this approach is that it effectively exposes the MarkLogic REST API to the end-user, which is a security vulnerability. (Application Builder directly exposes the REST API, too, with no middle tier.) MarkLogic does have a robust security model; it’s possible to lock down an application using that model, but there are a few reasons why I like doing so in the middle tier.

Role versus User Permissions

Consider the documents in which we store user profiles. In Demo Cat, I’m storing the user’s full name and a list of email addresses that the user has access to. In MarkLogic, we set document permissions with roles. Users who belong to a role that “update” permission may update a document. That means that to let my user (dcassel) update my profile, but not update another profile document, I would set up a “dcassel” role with update permission on the profile document. Only the dcassel user would get the dcassel role. This would work, but it isn’t in the spirit of how roles are intended to be used; rather, the intended use is that a role represents a group of users.

Note: there are certainly applications out there where roles were created for individual users, and that was done for legitimate good reasons. It’s not that you can’t or shouldn’t, but if you find yourself creating roles for individual users, it’s worth taking a look to see whether another way makes sense. 

There are other ways to limit who can update a profile document. For instance, I could create an extension specifically for doing so and have that extension check whether the current user matches the profile’s owner. However, doing that would still leave a backdoor open, in the sense that /v1/documents would need to be locked down.

Mindset

A big difference between limiting access to the MarkLogic REST API versus using internal security to limit what can be done with it is the mindset we use to approach it. If we directly expose the MarkLogic REST API to the end user, we’re in a black-list mentality — we need to figure out what to shut down. But if deny direct access to the REST API, we can approach it as a white list, deciding which features to expose, how, and to whom. To make an application secure, that makes more sense to me.

Mindset Part 2

Another note on mindset — MarkLogic is capable of hosting applications completely, without using a middle tier at all, but some customers don’t want that. They prefer to have a business logic tier with a hard line on a diagram connecting it to the database, rather than the database actually hosting the middle tier. (For anybody who read that and thought, “who would do that?”, when you have such a capable language running at the database level, you really can express all your business logic there. But I get it, you want more distance between the layers.) When you design a system with distinct business and database tiers, you’re not going to turn around and expose the database tier directly to the end user.

MarkLogic Security Still Useful

You should definitely not think, by reading the above, that MarkLogic security stops being useful when you introduce a middle tier. Far from it! With MarkLogic security, you can decide which groups of users (roles) can see which documents, then these restrictions are accounted for when performing searches. You can also easily prevent unregistered users from creating or deleting content, even in cases where your middle tier doesn’t know enough to decide what’s allowed. You have multiple levels at which to apply security, allowing you to focus on content (documents, directories, etc) or access methods (API).

Middle Tier Access Control

Okay, locking down the REST API makes sense, but does that mean we need to write custom endpoints for every touch on the database? For some systems, the answer is probably yes. Given that my goal is rapid application development, but trying to be on a path toward production applications, I’ll take the middle road. I want the client code (written in AngularJS) to be as portable as possible from one application to another.

Node.js hosts my static resources and exposes endpoints for functionality. For the sake of portability, my AngularJS code will make requests to REST API URLs. However, those requests go to node, where I can decide what to do with them. Consider this function:

Session management is done at the node layer, so if the user hasn’t logged in, no PUT requests are allowed. If the user has logged in and is PUTing a user profile, the profile has to be the one that belongs to the user. (I haven’t yet written code that will update the in-memory user profile based on the updates that have been sent.) The proxy() function simply relays the request on to MarkLogic, complete with credentials identifying the user.

You can see the rest of the middle tier at Demo Cat’s GitHub. You’ll see that GET requests are proxied right through (at the moment, I’m intending to change that); PUT and POST require the user to have logged in; DELETE requests are not supported at all; and a few /user/ endpoints govern logging in and out.

Locking Down MarkLogic’s HTTP App Server

The last step in controlling access to the REST API is to prevent people from bypassing your application and going straight to the REST API’s app server. This can be accomplished simply by having a firewall not allow external access, or you can set the MarkLogic HTTP app server’s address field to “localhost”, thus only allowing requests from the same server that MarkLogic is running on.

I’ve begun working on a Demo Catalog application recently that has a few purposes. My immediate purpose is to make our demos easier for our sales engineers to describe, find, comment on, and file bugs and RFEs against. If others find the app helpful as it develops, that’s great. But this project is also going to be my test-bed for some tech tools.

AngularJS

My team has been using AngularJS for a few months now, and a few projects in we’re really enjoying it. It takes a little time to ramp up, but then we’ve found we like the structure that AngularJS gives us. I’m starting to see re-usable pieces emerge and I expect to see more of that from this and other upcoming projects. We’ll be putting together directives and services that we often need while building MarkLogic demos.

MLJS

Speaking of Angular services, the architecture for this app involves talking to the MarkLogic REST API. I started down the path of making a brand new service to wrap the API, but I stopped and decided to see how far I could get with the MLJS Core libraries. I’ve been meaning to poke at MLJS for a long time.

I did run into a problem where MLJS’ pub/sub model didn’t really click with the promise approach that I’ve seen so much of in AngularJS, but Adam “Indefatigable” Fowler quickly put up an initial version of the MLJS Core that allows the use of promises instead (it’s not complete, but enough to proceed). Quickly, by the way, meant we chatted around 10:30 pm his time, and he sent me an email around 9:30 am the next day telling me I could check out the initial version. Awesome!

JSON

For all apps where I’m using JavaScript to talk to the REST API, I’ve used JSON for calls to /v1/search and other endpoints where it’s supported — although our documents are typically stored in XML, JSON’s a lot easier to work with on the client side.

What I haven’t really tried before is using JSON as the storage format for documents, but I’m trying that out here. I’m not looking to jump into the debate about JSON vs. XML — I like some aspects of each. Rather, I’m going to use this project as a test-bed for exploring the ability to store JSON in MarkLogic. It’s a feature we support, so I want to see how it works and whether I run into any limitations.

And some others…

I’m using the Roxy Deployer, which will come as no surprise to regular readers on this blog. I used Paxton Hare’s Yeoman MarkLogic generator to get the project started. I’m using Twitter Bootstrap for the layout, Karma and Jasmine for unit testing, and Grunt for a variety of tasks.

If you’d like to follow along, contribute, or set me straight as I figure things out, take a look at Demo Cat.