Before we go any further, let’s be clear that this is neither documented nor supported in the current version — and if it becomes official, the interface with the feature may very well change. If you are interested in this feature, please contact Stephen Buxton in Product Management and let him know.

With all that out of the way, let’s take a look.

ImageMagick is “a software suite to create, edit, compose or convert bitmap images”, as stated on its web site. If you have ImageMagick installed on your server when MarkLogic 5 starts up, you’ll see something like this line slide by in your ErrorLog.txt:

2012-01-09 17:10:41.557 Info: ImageMagick 6.7.4-4 2012-01-09 Q16 http://www.imagemagick.org

If you don’t have it installed, you’ll still see some reference to ImageMagick, but it will be a message indicating that it was not able to load ImageMagick.

Regular readers will have seen that I’m now running a site with the MarkLogic Express license: CurioVault.com. The site lets users upload information about collectibles they have, including when and where they got the item, an interesting story connected to it, and a picture of it. CurioVault is more about the stories than the pictures. It’s not Flickr — I don’t want the site to get bogged down with a bunch of multi-megabyte images. So, I set up the code so that I can specify a maximum size for an image in bytes, and automatically scale down any uploaded images that exceed that size. I also set up a maximum image dimension (in pixels) to use when scaling. Here’s the code that does it:

declare function item:scale-image($img as binary()) as binary()
{
  if (xdmp:binary-size($img) > $MAX-IMG-SIZE) then
    let $wand := magick:read-image(magick:wand(), $img)
    let $height := magick:get-image-height($wand)
    let $width := magick:get-image-width($wand)
    let $wand :=
      magick:scale-image(
        $wand,
        fn:min((fn:ceiling($width * $MAX-IMG-DIM div $height), $MAX-IMG-DIM)),
        fn:min((fn:ceiling($height * $MAX-IMG-DIM div $width), $MAX-IMG-DIM)))
    return magick:write-image($wand)
  else
    $img
};

The trickery with the 2nd and 3rd parameters to magick:scale-image() is to make sure the aspect ratio remains the same, while setting a maximum for either dimension. But simple as that, I pass in binary image that a user uploaded and when it comes back, I know that it won’t be huge. (People who do a lot of image manipulation may have suggestions for a better approach; such comments are welcome, but don’t let that distract you from celebrating this feature!) Previously, to accomplish this would have required calls out to MLJAM or a service.

This integration is so new that even the internal documentation is mostly references to ImageMagick’s documentation at this point. If you want to play around a bit, you can map the code above to the corresponding functions on ImageMagick’s documentation and probably figure out how to make some other kinds of calls.

Be sure to let Stephen know what you think.

Next up for CurioVault: letting users crop images at upload time.

I’ve collected lapel pins since 1984. In that time, I’ve gotten more than 150 and I’ve started to forget when I got some of them, where I got them, and the stories that went with them. Hence CurioVault.com. Give it a look!

The site runs on a computer I bought for $150 on Amazon Marketplace — a used box with Intel Core 2 Duo, 2GB of RAM (upgradable to 4GB) and a 500 GB hard drive. Not a very hefty machine, but good enough to get a small site online.

I’ve seen many examples of MarkLogic being a great solution for huge data sets. But the productivity advantages apply just as well when dealing with a small site. Great tools help on big jobs and little. Throw in the hope that a small site will one day grow up to be a big one, and I’m excited to be building a site for myself on MarkLogic.

Yes, we’re talking about Model-View-Controller. I’ve had a number of conversations with people about whether MVC makes sense in an XQuery world, and I wanted to share my reasons for thinking it does.

The conversation typically centers on the Model part of MVC. The argument against MVC in XQuery points out that a big part of the Model’s job is to get rid of the impedance mismatch between the way data is stored (perhaps rows and columns in a relational database) and the way it is used (perhaps Java objects). In XQuery, data is both stored and used in XML, so there is no mismatch, ergo a Model is overkill. That’s a valid point as far as it goes — after all, that is one of the benefits of working with XQuery in general and MarkLogic in particular.

When we started with our framework, we took a VC approach — we used Views to separate out the presentation logic, and we used Controllers to handle request inputs. However, we still used library modules to do the work of taking those request inputs and turning them into some data to present. We wanted to isolate that logic so that we could write unit tests. Before long, we recognized that we were, in fact, building Models.

So what role does a model play in XQuery? A simpler one. Once you leave the impedance mismatch problem behind, the benefit you are left with is encapsulation. Many of our applications need functions to register, log in, and log out users, along with updating passwords and identifying the current user. Each of these is a function that we store together in a library module, which we write unit tests for and can easily pull out of one project and put into another. That’s our user model. Likewise, many applications have some central concepts to them that lend themselves to building a model, even though there is no corresponding table, as we would have in the relational world.

Yes, this is a thinner version of how people think of a Model in the relational world. But there are benefits to having divisions between the parts of your application, and MVC is an approach that is familiar and (when used properly) has been useful.

Modules Database

1. Deploying code is a transactional update — there’s no need to worry that the server will be reading your source files while you’re in the middle of updating them.
2. Related to #1, if you’re working in a cluster and an application server is present on multiple instances, deploying to a modules database that spans those instances will let the application server on all instances update to the new code at the same time.
3. When you’re migrating an application from one server to another, you can use XQSync to move the source code as well as the data.
4. Using a modules database is actually required for CPF. In a database where you set up content processing, check the Configure tab for the domain — the evaluation context has to be a database.

File System

1. When you make a change in your code, there’s no deploy step. Just hit save and then refresh your browser (or however you are accessing results). Very simple.
2. You’ve got your code in a version control system like SVN or GIT (right?), and those expect to live on the file system. I guess you could set up WebDAV and do an “svn checkout …” right into a modules database, but that feels a little weird to me. Your code probably lives in a file system directory somewhere, and you interact with the version control system from there.
3. When deploying to an integration or production server, I’ve sometimes pointed the application server to a soft link, then update the soft link on the file system after setting up a directory with the new code. I think that addresses point #1 under Module Databases, but not #2 unless you do it on a volume that is visible to all instances.

What Do I Do?

Common practice among my colleagues seems to be use the file system while developing on your local box, then deploy to a modules database for integration or production servers. I’ve actually been using a modules database even for local work lately, because I’ve got an ant script that makes it simple and fast. Even with that, every now and then I forget to deploy.

Where does your code live? Any reasons I haven’t covered?

Direct

This is the simpler case, so I’ll take it first. Direct means that your XQuery code has the XML you want to build:

declare namespace blog = "http://davidcassel.net/blog";

<blog:example simple="true">
  <blog:pointless/>
</blog:example>

XML is a natural data structure for XQuery and this is a very simple way to construct it.

Computed

A computed element is less direct, but it has its benefits. Let’s take a look:

declare namespace blog = "http://davidcassel.net/blog";

element { xs:QName("blog:example") } {
  attribute simple { "true" },
  element { xs:QName("blog:pointless") } { }
}

Benefits

The direct method is more concise, so why would we use the computed? There are two reasons why I typically end up using them.

A Conditional Attribute

Sometimes you want to include an attribute only under certain circumstances. The trick is, this doesn’t work:

<blog:example { if ($condition) then simple="true" else () }/>

That’s not valid syntax — the attribute either has to be there or not, though its value can be the result of an XQuery expression. One way we can conditionally include an attribute is to build the element different in the if and else branches of the condition:

if ($condition) then
  <blog:example simple="true"/>
else
  <blog:example/>

As you can imagine, that can be a lousy way to go if the element has some complexity to it, for instance if it has a lot of attributes. This is one place where computed construction makes your life easier:

declare namespace blog = "http://davidcassel.net/blog";

element { xs:QName("blog:example") } {
  if ($condition) then attribute simple { "true" } else (),
  element { xs:QName("blog:pointless") } { }
}

Computing the Name

Another case where the computed version comes in handy is when the name of the element itself will be the result of a computation. Let’s suppose you have a map that you want to turn into a set of elements.

<root>{
  for $key in map:keys($map)
  return element { $key } { map:get($map, $key) }
}</root>

There’s really no other way to do this case, but it’s simple to do this way.

Any other cases you can think of where computed elements are the easier way to go?

Anyway, as we set up our XQuery standards, we chose to rely on Douglas Crockford’s JSLint to set the standards for our JavaScript code. I found the quote below in a README file as part of JSLint and I thought it was worth sharing.

The place to express yourself in programming is in the quality of your ideas,and the efficiency of execution. The role of style is the same as in literature. A great writer doesn’t express himself by putting the spaces before his commas instead of after, or by putting extra spaces inside his parentheses. A great writer will slavishly conform to some rules of style, and that in no way constrains his power to express himself creatively. See for example William Strunk’s The Elements of Style [http://www.crockford.com/wrrrld/style.html].

This applies to programming as well. Conforming to a consistent style improves readability, and frees you to express yourself in ways that matter. JSLint here plays the part of a stern but benevolent editor, helping you to get the style right so that you can focus your creative energy where it is most needed.

As someone who writes both code and prose, I could relate well to both aspects of what he was saying. I keep my copy of Strunk pretty close at handy.

Reading through that I was left with a couple questions. In particular, I wanted to know how smart the lexical analysis is. Turns out its pretty good (no surprise)!

A Simple Query

Time for some code. Here’s a simple module that we hit to see whether we have a query or an update.

<h1>{
  if (xdmp:request-timestamp()) then "query"
  else "update"
}</h1>

As the documentation tells us, xdmp:request-timestamp() will return a value if and only if we’re doing a read-only query. Sure enough, when we run this module, we get <h1>query</h1>.

A Simple Update

if (0 = 1) then xdmp:document-insert("/foo.xml", <foo/>) else (),
<h1>{
  if (xdmp:request-timestamp()) then "query"
  else "update"
}</h1>

This one announces itself as an update. Despite the fact that the xdmp:document-insert() will never be called, the server sees this as a potential update. We’re going to use this expression as our trigger to put us into update mode.

A Library Update

Let’s introduce a simple library module.

module namespace lib1 = "lib1";

declare function lib1:f1()
{
  if (0 = 1) then xdmp:document-insert("/foo.xml", <foo/>) else ()
};

declare function lib1:f2()
{
  <harmless/>
};

The lib1:f1() function has our trigger. Let’s see what happens if we import the module but don’t call the function.

import module namespace lib1 = "lib1" at "lib1.xqy";

<h1>{
  lib1:f2(),
  if (xdmp:request-timestamp()) then "query"
  else "update"
}</h1>

That’s a query — MarkLogic Server is smart enough to see that while there are update expressions in the library module, there are no calls that invoke that expression. Now let’s put in a call to f1() — our trigger function — and see what happens.

import module namespace lib1 = "lib1" at "lib1.xqy";

<h1>{
  lib1:f1(),
  if (xdmp:request-timestamp()) then "query"
  else "update"
}</h1>

You guessed it, now we have an update function.

Apply a Test

One last thing to try: can we fool the analysis with xdmp:apply()?

xquery version "1.0-ml";

import module namespace lib1 = "lib1" at "lib1.xqy";

<h1>{
  xdmp:apply(xdmp:function(xs:QName("lib1:f1"))),
  if (xdmp:request-timestamp()) then "query"
  else "update"
}</h1>

What do we get this time? The analysis shows that this is a query statement, but when the server actually runs the code, it finds itself doing an update. That breaks the rules — a query runs at a particular timestamp without locks. An update without locks would obviously cause problems. So what do we get running the query above?

500 Internal Server Error

XDMP-UPDATEFUNCTIONFROMQUERY: xdmp:apply(xdmp:function(xs:QName("lib1:f1"))) --

Cannot apply an update function from a query

Kaboom. Check for this when using xdmp:apply().

Columbo Moment

Oh and, there is just one more thing…. If you want to tell MarkLogic Server explicitly that a module should run in update mode without using a trigger expression like the one above, you can put this into your module:

declare option xdmp:update "true";

You might do that to avoid an XDMP-UPDATEFUNCTIONFROMQUERY while using xdmp:apply().

(1,3,2)[.] -> 1
(1,2,3)[.] -> 1 2 3
(2,3,4)[.] -> no result
why?

To find the answer, we need to think about what’s happening in the brackets. As discussed in one of my earlier posts, the expression in brackets gets evaluated once for each item in the sequence. The result of that evaluation on each item determines whether that item gets selected.

My initial thought was that the answer to this question was pretty simple: the “.” refers to “self”, the current item being evaluated, and the evaluation used the value of the current item. That gets us a number, which when used as a filter expression for a sequence, indicates the index of the item we want to select. We can expand the example above to this:

(1, 3, 2)[fn:data(.) eq fn:position()]

Now it’s pretty clear why we get just (1) as our answer — for this case, 1 is the only item where the value matches the position.

But wait, the plot thickens! XQuery allows us to throw mixed types of items into a sequence. What do we get from this:

(1, <a/>, 3, 2, “blah”)[.]

The answer is (1, <a/>, 3, “blah”). The expansion I gave above doesn’t hold here, because we can’t compare fn:data(<a/>) and fn:position() — I get an error about trying to compare an xs:untypedAtomic(“”) to a number. So what really happens? To find out, I turned to MarkLogic’s own John Snelson, whose familiarity with the XQuery spec is a good deal deeper than mine. Here’s what I found out:

The implicit operation in the predicate is more like this:

typeswitch(.)
case xs:decimal | xs:float | xs:double return . eq position()
default return fn:boolean(.)

The fn:boolean() function returns true for a sequence containing a single node – so the constructed node in your last example passes the predicate. Similarly, a non-zero length string returns true from fn:boolean().

So there we have it. If the value is a number, it’s compared to the position. If it’s not a number, we check whether the value is, to borrow from Douglas Crockford, “truthy”. More precisely, XQuery finds the Effective Boolean Value for the context item and uses that to decide whether the item in the sequence should be returned by the filter expression.

<search-criteria id="1">
  <param key="keywords">food</param>
  <param key="project-name">Important Work</param>
  <param key="date">2011-12-31</param>
</search-criteria>

As you can see, the function is creating generic element names (“param”) with an attribute designed to give it meaning. And here’s the function:

declare function local:create-search-criteria($params as map:map) as element(search-criteria)
{
  let $form-elements := (
    "project-name",
    "date"
  )  

  let $keywords as xs:string := (map:get($params, "keyword"), "")[1]
  let $grant-id as xs:unsignedLong := local:get-next-id()

  return
    <search-criteria id="{ $grant-id }">
      <param key="keywords">{ $keywords }</param>
      {
        for $key in $form-elements
        let $value := map:get($params, $key)
        return
          <param key="{ $key }">{ $value }</param>
      }
    </search-criteria>
};

Why Not?

It’s certainly fair to ask, “Why not do it this way?” After all, we can still use XPath to select a specific part of the XML. For instance:

local:create-search-criteria($params)/param[@key = "date"]

That’s good enough to get the date parameter element.

While that’s true, I can think of two arguments against doing it this way, one specific to MarkLogic, one that is perhaps a purist’s argument.

Meaningful Structure

The first reason I would avoid this structure is simply that there is a way to express it that is more concise without being unclear. As with writing, if you can say the same thing more briefly without losing clarity, then it is probably a good idea. In this situation, the param element name does not contribute to our understanding of the content.

Indexes

In MarkLogic, there is no way to set up a range element index that is dependent on the value of an attribute. For instance, we could not set up a range element index on param that only had the values where the key parameter is “date”. A range element attribute index doesn’t solve the problem, because that indexes the values in the attribute itself.

Revised

Let’s start with how we might like the XML to look:

<search-criteria id="1">
  <keyword>food</keyword>
  <project-name>Important Work</project-name>
  <date>2011-12-31</date>
</search-criteria>

Now our structure is simple and concise and it will be easy to set up some useful indexes.

For the function, we can let the $param map govern the output structure.

declare function local:create-search-criteria($params as map:map) as element(search-criteria)
{
  <search-criteria id="{ local:get-next-id() }">
  {
    for $key in map:keys($params)
    return element { $key } { map:get($params, $key) }
  }
  </search-criteria>
};

Or if we prefer to control which fields get pulled out of the map, that’s a simple matter of setting up a sequence, similar to the original.

declare function local:create-search-criteria($params as map:map) as element(search-criteria)
{
  <search-criteria id="{ local:get-next-id() }">
  {
    for $key in ("keyword", "project", "date")
    return element { $key } { map:get($params, $key) }
  }
  </search-criteria>
};

let $normal :=
  try {
    xs:int($value)
  }
  catch($e) {
    $value
  }

Yes, MarkLogic extends XQuery to add try/catch, and this is a tremendously useful feature, when used in the right place. In this situation, I think the person who wrote the code either didn’t know how to write this a different way, or didn’t realize that try/catch is typically slower. Here it is, properly done:

let $normal :=
  if ($value castable as xs:int) then
    xs:int($value)
  else
    $value

Does it matter? To find out, let’s ask the profiler. I made a call to the function in CQ and clicked the profiler button, once for each case. Of course, other things happen in the function. In fact, this function calls the above expression over 800 times for a real-world case.

• try/catch — 0.29 seconds
• if/then — 0.029 seconds

That’s an order of magnitude difference.

Moral of the story: as with most languages, exception handling is meant for exception cases. If the catch gets invoked, it should be an unusual case that can’t reasonably be handled by the normal flow of the program.