Using DITA for Genealogical Data

I’ve been working on putting my family history together for the last few years.  Most of the genealogical applications have some pretty nice features, but none seemed to have all of the features I wanted.  I wanted the ability to manage all of the summary information and relationships (all applications do this), but also cross reference the factual data with individual biographies.  And, I want to be able to display the information in different ways and formats – not just the ones supported by any particular application.

I started looking at the format that most genealogy programs store the data into.  With few exceptions, they all use GEDCOM, or GEnealogical Data COMmunication.  The standard was developed by the Church of Jesus Christ of Latter-day Saints as a means of creating a portable data format to express information about individuals, families and sources (bibliography).

GEDCOM is line-delimited field format that identifies the start of a new record with the number 0.  Fields within a record are identified with an incremented number.  For example a first level field line would start with the number 1.  A subfield (e.g, the given name of a person’s full name) would start with the next highest number. The following is an example record for an individual

  0 @1@ INDI
    1 NAME   Robert Eugene/Williams/
    1 SEX   M
    1 BIRT
      2 DATE   02 OCT 1822
      2 PLAC   Weston, Madison, Connecticut
      2 SOUR   @6@
        3 PAGE   Sec. 2, p. 45
        3 EVEN   BIRT 
          4 ROLE   CHIL
    1 DEAT
      2 DATE   14 APR 1905
      2 PLAC   Stamford, Fairfield, CT
    1 BURI
      2 PLAC   Spring Hill Cem., Stamford, CT
    1 RESI
      2 ADDR   73 North Ashley
        3 CONT   Spencer, Utah UT84991
      2 DATE   from 1900 to 1905
    1 FAMS   @4@
    1 FAMS   @9@

 

Other than the line/sequencing delimiters, the data structures are pretty free form, and is parser dependent.  Even the field names, outside the common set supplied by GEDCOM are parser dependent.  So if you use one genealogy tool, it can understand these fields, but if you try to load it in another, it blows up.  Gah!  Add to that, GEDCOM just isn’t that great for handling rich content like pictures in a biography.

This sounds like a job for XML. 

So the first question I had to address is how to model this.  I’ve looked at some the of GEDCOM XML sites, and they suffer from the same problems as the text data structure do.  Just not enough rich data. 

The answer I came up with was to use DITA, which has several things going for it:

1. I can easily mimic GEDCOM’s data structure with a specialized map
   
2. I can extend the model to support other potentially valuable metadata
   
3. I can easily model rich biographical content as a topic specialization
   
4. DITA’s numerous linking mechanisms work well for the various types of links I would need:  internal references within a map, rel-tables, cross-references, external hyperlinks to third-party websites and content.

The first thing I did was to model and create a map specialization that mimics the GEDCOM data.  For the sake of brevity, I’ll show a sample of a specialized map.  If you want more information, ping me:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE familytree SYSTEM "file:/opt/dita/1.2/dtd1.2/genealogy/dtd/familytree.dtd">
<familytree>
    <title>Schmoe Family Tree</title>

    <individual id="I1" keys="I000001" gender="male">
        <vitals>
            <personname>
                <firstname>Joseph</firstname>
                <firstname type="nickname">Joe</firstname>
                <middlename>Aloysius</middlename>
                <lastname>Schmoe</lastname>
                <generationidentifier>III</generationidentifier>
            </personname>
            <birth>
                <date>
                    <day>1</day>
                    <month>1</month>
                    <year>1968</year>
                </date>
                <location>
                    <placename>The Stork Factory</placename>
                    <addressdetails>
                        <locality>Anytown</locality>
                        <administrativearea>Anystate</administrativearea>
                        <country>USA</country>
                    </addressdetails>
                </location>
            </birth>
        </vitals>
    </individual>
    <individual id="I2" keys="I00002" gender="male">
        <vitals>
            <personname>
                <firstname>John</firstname>
                <firstname type="nickname">Jack</firstname>
                <middlename>Michael</middlename>
                <lastname>Schmoe</lastname>
            </personname>
            <birth>
                <date><day>1</day><month>1</month><year>1948</year></date>
            </birth>
        </vitals>
    </individual>
    <individual id="I000003" keys="I000003" gender="female">
        <vitals>
            <personname>
                <firstname>Jane</firstname>
                <middlename></middlename>
                <lastname type="maidenname">Doe</lastname>
                <lastname type="marriedname">Schmoe</lastname>
            </personname>
            <birth>
                <date>
                    <day>1</day>
                    <month>1</month>
                    <year>1947</year>
                </date>
            </birth>
        </vitals>
    </individual>
    
    <family id="f1" keys="F1">
        <familymeta>
            <marriage>
                <date>
                    <day>1</day>
                    <month>1</month>
                    <year>1967</year>
                    
                </date>
            </marriage>

        </familymeta>
        <child keyref="I000001"/>
    </family>
    <familyreltable>
        <record>
            <indi>
                <personref keyref="I00001"/>
            </indi>
            <famc>
                <familyref keyref="F1"/>
            </famc>
            <fams/>
        </record>
        <record>
            <indi>
                <personref keyref="I00002"/>
            </indi>
            <famc/>
            <fams>
                <familyref keyref="F1"/>
            </fams>
        </record>
        <record>
            <indi><personref keyref="I00003"/></indi>
            <famc></famc>
            <fams>
                <familyref keyref="F1"/>
            </fams>
        </record>
        
    </familyreltable>
</familytree>

The Butterfly Effect of Oracle’s Lawsuit Against Google

No one knows with any degree of certainty what the outcome will be from Oracle’s patent infringement lawsuit against Google.  While the impetus of the lawsuit is largely focused on Dalvik, Google’s mobile VM based on Java, the consequences of legal action will likely reverberate through the larger Java world. 

I don’t have a dog in this fight.  Yet. On the one hand, Oracle appears to be hell bent on reversing Sun’s decision to release Java to the open source community, according to an InfoWorld article:

It's no secret that Larry Ellison wants to make money from Java, something Sun's execs, whom Ellison held in contempt, was never able to do. It may be that Oracle wants nothing more than a cut of Google's Android revenue -- IDC's Will Stofega told Bloomberg News that the case will probably end with Google agreeing to pay to license Oracle's patents.

On the flip side, it sure looks like Google didn't do any favors for itself either. 

Whatever the case, there are potential far reaching implications of Java in the open source world.  For one, it will reinvigorate the debate around software patents. More specific to my interests here, XML was, and is, heavily influenced by Java and open source. Think Xerces and Xalan, Saxon, FOP, Ant, and more recently Calabash and Calumet.  It also factors in to related technologies like the APIs  for eXist and MarkLogic, and fuels reference implementations of standards like DITA (see the DITA Open Toolkit).

My immediate reaction is the cat’s already out of the bag: Java is already open source, which means nothing will change for XML technologies.  For now.  In the short term, it’s very likely the only ones affected are Android application developers.  Longer term, however, Oracle’s behavior might well impact a wide array of technologies, including XML, by deterring developers from using Java in the first place.  That would be a huge detriment to XML across many technologies and industries.  Let’s hope it doesn’t get to that point.

Technorati Tags: XML,Oracle,Open Source

Cancer Sucks

My Grandparents died from it. My mother had it (in remission), but now she needs to have a full mastectomy because a genetic marker indicates she's at higher risk to get it again (she has surgery on March 7). And now, a friend and colleague of mine has it. This guy is healthier than 95% of other men his age, has like zero body fat, and is always encouraging everyone to ride with him (more like follow from a long distance). That's just hard to take in and process. While he's the one that has his world rocked, cancer also has a nasty side effect on family and friends. Put it bluntly: Cancer sucks.

Few other spoken words, irregardless of language or dialect, can evoke as much emotion as 'cancer'. Even as scary as influenza and other diseases like AIDS are, and as much devastation they can bring to an individual, a family, a community, and even the world, there's something about cancer that is so viscerally scary to us. For me it's because cancer is, in my non-medical, simpleton view, a mutation of cells. It's not like a virus or an infection where a foreign organism is making you sick, and you can take medication to kill off the nasty invaders - it's your own cells that, for some reason have started going haywire. That's just downright frightening to me. And that cancer is so seemingly random and unpredictable just makes that much more hard to take - you could be in the best possible physical shape and still get it.

Science and medicine have come so far when it comes to curing many forms of cancer. Many people have a good chance of living long, healthy lives if they catch the disease early. But often they have to go down through the depths of hell physically and emotionally to get to the finish line: cancer-free. Yet, they keep their eyes on the prize.

I'm pretty sure my friend will beat this. His odds are really high, and his outlook on life and his current illness are positive, which also increases his chances.

If nothing else, it's a reminder that life is too short, so live it to its fullest. It's so easy to become burdened with frustration, stress, angst, even hatred. In other words, we lose sight of the forest within the trees. I'm probably the most susceptible to this.

In short: Work hard. Play harder. Laugh more. Celebrate with your family and friends. Find something everyday that is good. A sunrise or sunset, a laugh with your kids, a hug or a kiss with your spouse or significant other, a walk in the park with your dog. It's your life - own it.

Apple to Release Tablet PC Today: e-Books are One Target Market

Apple is releasing their tablet PC today.  According to NPR, they plan to include an e-reader.  As I mentioned in a previous post, I though that the tablet was the perfect medium for e-books, and would supplant the Kindle, Nook and other current e-book devices.  Only time will tell for sure.

The move by Apple is brilliant.  It already has a first class distribution model with iTunes pushing out applications and music for the iPhone and iPod.  e-Books are just a natural progression. 

From an XML publishing standpoint, EPUB is a relatively easy format to render. The DocBook XSLT stylesheets has an EPUB format built in.  I'm not aware of one for DITA yet, but I can't imagine that it would be too far behind, or difficult to build.

Tablet PCs - What does this mean for e-Readers?

HP previewed their new tablet PC at the 2010 Consumer Electronics Show.  Apple will release their version later this spring.  Sounds to me like this could supplant the Kindle, Nook and other e-Reader devices.  It might also be the impetus for EPUB distribution.  e-Readers are nice, but they're one trick ponies. 

From an education publishing perspective, this could really open the door to new revenue models for schools at all levels to distribute published content.  It also opens the door for new models for authors writing content. 

More on these ideas later.

Fun With XProc

I've been so busy with clients over the last 6 months that I haven't had much time to tinker with XProc much. I took the Thanksgiving holiday week off with the hope of having a little time to dabble with the language. Up until yesterday, I didn't open my computer once (has to be a new record) since we were busy with other things. A side note: If you're in Denver before February 7th, I highly recommend you see the Ghengis Khan exhibit at the Museum of Nature and Science.

As I often do, I had already done some preliminary reading beforehand. James Sulak's blog is a must-read. Another very useful and informative website is from EMC: "XProc: Step By Step", originally authored by Vojtech Toman. Even the W3C specification is generally helpful.

The biggest hurdle for me was to stop thinking of XProc working in the same way I think of Ant. While Ant does process XML content, it isn't the tool's principle focus - Ant was principally designed as Java implementation of MAKE tools. For that purpose, Ant has become the de facto standard. Before XProc was conceived, many of us used Ant as a way to control the sequencing of complex XML publishing pipelines. I worked on XIDI, a DocBook-based publishing system at HP, which was principally based on Ant scripts; the DITA Open Toolkit is an Ant-based build environment. For the most part Ant works admirably, but there are limitations. The biggest limitation is the xslt task's static parameter declarations, and the indirect nature by which parameter values are passed to an XSL Transformation through property values. It works, but it can get kludgy pretty fast for complex stylesheets. More importantly, Ant is primarily a developer's tool that acts like a Swiss Army knife that has a tool for just about every purpose. Most of these tools work very well for very specific tasks, but they aren't intended to perform specialized tasks. For that, you'll need to create custom Ant tasks or use other specialized tools. XProc is one of these specialized tools that is designed specifically for XML processing.

So the biggest conceptual difference to grok in XProc (I like this…) is how steps are connected together to form a complete pipeline process. Rather than using target dependencies and explicit target calls like you do in Ant, XProc uses the concept of pipes to connect the output of one step to the input of the next step. It's very much like Unix shell or DOS command line pipelines. For example:

ps -ax | tee processes.txt | more

Since many steps (including the p:xslt step) can have more than one input and one or more outputs (think of xsl:result-document in XSLT 2.0) we need to explicitly bind uniquely named output streams to input streams of subsequent steps. It's very analogous to plumbing, and another way that XProc is different than Ant: Ant's tasks are very dependent on the file system to process inputs and outputs; XProc pipelines are in-memory input and output streams until you explicitly serialize to the file system.

With this I was able to create my first "real" XProc pipeline to generate Doxsl output. Here it is:

<p:declare-step name="doxsl" type="dxp:generate-doxsl-docs"
psvi-required="false"
xmlns:p="http://www.w3.org/ns/xproc"

    xmlns:dxp="urn:doxsl:xproc-pipeline:1.0">  
 <p:input port="source" kind="document" primary="true"   
     sequence="false" />   
 <p:input port="parameters" kind="parameter" primary="false"   
     sequence="true"/>   
 <p:output port="result" primary="true" sequence="false" >   
     <p:pipe step="transform" port="result"/>   
 </p:output>   
 <p:output port="secondary" primary="false" sequence="true" >   
     <p:pipe step="transform" port="secondary" />   
 </p:output>   
 <p:option name="format" select="'dita'"/>   
 <p:choose name="select-stylesheet"> 
     <p:when test="$format='dita'">  
         <p:output port="result" primary="true"   
             sequence="false" >   
             <p:pipe step="load-dita-stylesheet"   
                 port="result"/>   
         </p:output>   
         <p:load name="load-dita-stylesheet">   
             <p:with-option name="href"   
                 select="'../../dita/doxsl.xsl'" >   
                 <p:empty/>   
             </p:with-option>   
         </p:load>   
     </p:when>   
     <p:when test="$format='html'">   
         <p:output port="result" primary="true"   
             sequence="false">   
             <p:pipe port="result"
                step="load-html-stylesheet"/>   
         </p:output>   
         <p:load name="load-html-stylesheet">   
             <p:with-option name="href"   
                select="'../../html/doxsl.xsl'"/>   
         </p:load>   
     </p:when>   
 </p:choose>   
 <p:xslt name="transform">   
     <p:input port="source" >   
         <p:pipe step="doxsl" port="source"/>   
     </p:input>   
     <p:input port="stylesheet" >   
         <p:pipe step="select-stylesheet" port="result"/>   
     </p:input>   
     <p:input port="parameters">   
         <p:pipe port="parameters" step="doxsl"/>   
     </p:input>   
     <p:with-param name="debug" select="'true'"/>   
 </p:xslt>   
</p:declare-step>

Here's a diagram, built with EMC's XProc Designer.  This tool is a great way to visualize and start your XProc scripts:

Essentially, I used the p:declare-step declaration so that I can declare it as a custom step (dxp:generate-doxsl-docs), which will allow it to be integrated into other pipelines. It has one option, format, which is used to specify which output format to generate (for Doxsl, 'html' and 'dita' are supported). The first step ("select-stylesheet") evaluates the format option and loads the appropriate stylesheet into the step's "result" output stream. This is used by the second step's ("transform") stylesheet port. The transform's source file (the XSLT stylesheet to be documented) is bound to the root step's source port, as is the parameters port. I also set the stylesheet's 'debug' parameter to true to inject output to the "transform" step's result port.

All of this is done in memory and not serialized to the file system. This is intentional so that other pipelines can integrate this custom step.

I've tested this with Calabash. I still need to evaluate with Calumet.

Right now, these are baby steps. I think that XProc has a lot of potential. I think the next big task is to consider an XProc implementation for DITA XML processing.

DITA’s New Keys and the 80/20 Rule

Have you ever used the lorem() function in Microsoft Word? How about the rand() function? Do you know all the function keys? Most of us have used Microsoft Word for countless years and don’t know about all of the “hidden” functionality that it offers. Chances are, you’ll know a few of these, but you won’t know all of them simply because you’ve never needed them. Many of these functions are extremely powerful utilities that make Word a versatile application beyond a standard formatted text editor. But they’re available if you ever have the need.

The same is true with some of the new functionality being made available in the forthcoming DITA 1.2 release currently being worked on. Of particular interest is the introduction of keys. Keys provide a way for authors to create addresses to resources through the use of a named identifier rather than to a specific URI pointer. In other words, I can create an easy-to-remember key, like “ms-word-functions” that actually resolves to a URL “http://support.microsoft.com/kb/211982” and link to this URL using the key name in my DITA topic.

Here’s an example of how it works. In my map, I define a topicref and set the keys attribute with an identifier. I also set my href to the physical location of the resource I want to reference.

<map>
    <topicref keys="ms-word-functions" 
        href="http://support.microsoft.com/kb/211982"
        scope="external"/>
</map>

In my topic file, I can reference the key that's defined in my map:

<topic id="my.topic">
    <title>SampleTopic</title>
    <body>
        <p>
            Lorem ipsum dolor sitamet,  
            consectetuer adipiscing elit. Maecenas
            porttitor conguemassa. Fusce posuere, agna 
            sed pulvinar ultricies, purus
<xref keyref="ms-word-function">lectus</xref>
            malesuada libero, sit amet commodo magna eros
            quisurna.
        </p>
   </body>
</topic>

Now, when the topic is rendered, it will resolve itself to the Microsoft URL defined in my map. Pretty cool stuff. And powerful too. This has many potential uses: localizers can create translated versions of a resource using the same key reference and resolve the link to a locale-specific version of the reference. Consumers can be directed to different resources based on their profile or context within a website.

From an authoring perspective, there's another neat user story: I can reference a "yet-to-be-determined" resource via a key, and when that resource has been created, the key's definition in the map file will resolve the key reference.

Technically, a key definition doesn't need to be reside directly in the map that references that topic. It can live in an "ancestor" map that pulls in the topic indirectly by way of the map referencing that topic. In fact key values can be overridden: Let's assume that I define a key, called "company-website" in Map A that points to "www.company-a.com", and in Map B, I define the same key as"www.company-b.com". Map B also references Topic-1.dita which contains a keyref to "company-website". Map A references Map B. When the Topic-1.dita is rendered in the context of Map B as the primary map, the keyref will resolve to"www.company-b.com"; when Map A is the primary map, the same topic willreference www.company-a.com.

• Map A
  Key: company-website = "www.company-a.com"
  • Map B
    Key: company-website = “www.company-b.com”
    • Topic-1.dita
      keyref: company-website
      resolves to: www.company-a.com
• Map B
  Key: company-website = “www.company-b.com”
  • Topic-1.dita
    keyref: company-website
    resolves to: www.company-b.com

With all great power comes even greater responsibility. Any time a topic makes use of a key reference, that topic is explicitly binding itself to a map(or many maps), meaning that a topic is no longer a unit of information that is completely independent of any particular context in which it is assembled into. You could make the argument that any reference defined in a topic to an external resource (e.g., an image or a cross-reference to another topic) by definition creates a dependency on that topic. And arguably, the referenced (the endpoint) resource is unaware of the object that is referencing it, regardless of whether it's a topic reference or a cross-reference. But there is an additional dependency in the case of keys: Any map that references a topic with a key reference must define the key. So in a sense, not only does the map (or an ancestor map) need to know about the topic, it needs to discover what the topic is about, specifically related to any key references it points to. Consequently, somewhere along the line, at least one map must define the keys used by a topic.  Did you get all that?  Imagine what your XML authoring tools, CMS systems, and rendering platforms will need to do to manage this.

This is pretty sophisticated and powerful functionality.  But the question is, do you need to use keys and keyrefs in order to use DITA?  More importantly, will your tools need to support keys to take advantage of DITA's other capabilities?  The short answer is no.  In fact, I would expect that keys/keyref -enabled DITA support is still a way off for most DITA-enabled tools.  Nevertheless, you can still use DITA with the current tools and get most, if not all, of what you need.  Just like Microsoft Word with features like MailMerge, keys and keyref will be there if you need them, but chances are, you can get by without them for most content without ever knowing you missed it.

Finally,  the possibility of defining indirect links has opens the door to many different possibilities for dynamically driven, profile- and locale-specific content.  This is very cool stuff - the kind of thing XML folks like me get excited about.  But from a practical standpoint, there are potential downsides too.  Keys and key references add another layer of complexity to planning the authoring, deployment and management of DITA content.  In reality, most tools aren't ready for this complexity just yet.  So while the the standard is ahead of the game, the rest of the industry will be playing catch up.  Still, Ride the Wave.