David Cassel

void DayOfWeek::map(TupleIterator& values, Reporter& reporter)
{
  marklogic::DateTime v;
  time_t iVal;
  struct tm timeval;
  int dow;
  for(; !values.done(); values.next()) {
    if(!values.null(0)) {
      values.value(0, v);
      iVal = v;
      localtime_r(&iVal, &timeval);
      dow = timeval.tm_wday;
      counts[dow] += values.frequency();
    }
  }
}

When this function ends, the counts array will have values representing this particular stand. Separately, the same method will be called for every other stand in the database, on as many MarkLogic nodes as have forests for the database. Thus, we end up with a bunch of different DayOfWeek objects, each holding a set of counts.

Reduce

To consolidate the DayOfWeek UDF objects, we enter the reduce phase. Every time we call this function we take two of these objects and consolidate into one, in this case combining the counts arrays of each. In the diagram above, the values from Forest 1’s two stands will get reduced together, forest 2’s stands will get reduced together, and forest 1 and 2’s values will get reduced together. Eventually these will be combined with the values from host B, resulting in a single, consolidated set of values.

void DayOfWeek::reduce(const AggregateUDF* _o, Reporter& reporter)
{
  const DayOfWeek*o = (const DayOfWeek*)_o;
  for (int i=0; i<7; i++) {
    counts[i] += o->counts[i];
  }
}

Since we reduce two into one with each call, you can imagine that this function gets called one less time than we called map(), ultimately resulting in a single DayOfWeek object with a single set of counts values.

Finish

Once the reduce stage has gotten down to a single object, it’s time for that object to format the results in the way MarkLogic expects to receive them. MarkLogic will build an XQuery map:map() object from the results, so the finish method writes out its data in a way that can be used to build a map.

void DayOfWeek::finish(OutputSequence& os, Reporter& reporter)
{
  reporter.log(Reporter::Info, "DOW finish()");
  char key[2];
  os.startMap();
  for (int i=0; i<7; i++) {
    sprintf(key, "%d", i);
    os.writeMapKey(key);
    os.writeValue(counts[i]);
  }
  os.endMap();
}

To do this, we use an OutputSequence, which knows how to build a map from a set of values. We tell it to start the map, then we alternate specifying a key and a value. We end the map, and MarkLogic takes it from there. I also threw in another reporter.log() call for tracking. These reports will end up in the usual MarkLogic ErrorLog.txt.

Encode and Decode

If your database has forests on multiple nodes, then at some point MarkLogic will need to transfer UDF objects between nodes in order to reduce them. To accomplish this, a UDF must implement the encode and decode methods, showing how to serialize and deserialize the data. For the DayOfWeek UDF, this just means writing and reading the contents of the counts array.

void DayOfWeek::encode(Encoder& e, Reporter& reporter)
{
  for (int i=0; i<7; i++)
  e.encode(counts[i]);
}

void DayOfWeek::decode(Decoder& d, Reporter& reporter)
{
  for (int i=0; i<7; i++)
  d.decode(counts[i]);
}

This task is made easier by the Encoder and Decoder objects passed into the methods. These classes are provided by MarkLogic. All I need to do is call the encode and decode methods on the respective objects in the same order.

Making the UDF Available

There are additional details I could get into with the UDF implementation, but I want to keep this example pretty simple. Let’s move on to how to use the UDF. We first need to tell MarkLogic about it. There are a few steps in this process.

Registering a Name

The first step happens in the code, where we specify how this UDF should be registered. You’ll find examples of this at the end of SamplePlugin.cpp. Here’s what I did for DayOfWeek:

extern "C" PLUGIN_DLL void
marklogicPlugin(Registry& r)
{
  r.version();
  r.registerAggregate<DayOfWeek>("day-of-week");
}

This associates a name (“day-of-week”) with the class (DayOfWeek). We’ll use this name when we actually want to call the UDF.

Building

Naturally we would compile and build while developing the UDF, to catch any syntax errors. We can use the MarkLogic-provided Makefile to do this. Simply calling make without specifying a target will build a dynamically-loaded library based on the platform where the build is done (.dll, .so, .dylib) and zip it up along with a manifest expected by the MarkLogic plugin loader.

Installing

Next, we use Query Console to tell MarkLogic about the ZIP file, using the XQuery function plugin:install-from-zip():

xquery version "1.0-ml";
import module namespace plugin = "http://marklogic.com/extension/plugin"
  at "MarkLogic/plugin/plugin.xqy";
plugin:install-from-zip("native",
  xdmp:document-get("/Users/dcassel/project/plugin/day-of-week.zip")/node())

MarkLogic will read in the ZIP file and copy the contents to a safe place where it will be able to find it (on my Mac, it was ~/Library/Application\ Support/MarkLogic/Data/Lib/native/day-of-week/; on Linux /var/opt/MarkLogic/Lib/native/day-of-week.) At this point, the UDF is considered a native plugin and is available to be used — MarkLogic will see it when the UDF is called without any need to restart MarkLogic itself.

Calling

Last but not least, we can call the UDF.

cts:aggregate(
  "native/day-of-week", 
  "day-of-week", 
  cts:element-reference(xs:QName("postedTime"), "type=dateTime"),
  (),
  (),
  "soccer"
)

Here I’ve called the cts:aggregate() XQuery function, which is the gateway to using UDFs. I’ve specified which native plugin to use and the name of the UDF that I want. The cts:element-reference() call is a new way to specify what I want to target, in this case the element range index on postedTime, which has the dateTIme type. In the last parameter, I threw in a “soccer” query, so the day-of-week calculation will take place on postedTime values in fragments that match the word query “soccer”.

Running this in Query Console, I get back a map:map(), as you would expect from looking at the finish method above. The results map from day-of-week code (0 = Sunday) to the count of dateTimes that fell on that day of the week. I’ve got some future posts in mind where we’ll look at what to do with the map data once you have it.

Your Turn

The ability to define UDFs, along with the newly provided build-in aggregation functions, opens up new ways to look at your data. I’m curious what you come up with. Leave a comment with an idea for a useful UDF!

(1) Windows: C:\Program Files\MarkLogic\Samples\NativePlugins\; Mac: ~/Library/MarkLogic/Samples/NativePlugins/