Repurposing XML statistics for InDesign automation

22 03 2010

Well! I’ve been working on a lovely statistics prototype. Using the latest jQuery, theme rolled to match my employer’s corp ID, and the awesome free trial of Fusion Charts (along with a bit of php and xsl mastery), I’ve made a pretty cool stats prototype website.

All the graphs OR table views are rendered from XML files containing the institution’s statistics, both views support clickable drill downs and the whole interface is slick as, but it stops at on-screen graph/table viewing.. Even though the UI is rich and fun to use (accordion, nice jQuery forms and tabs), I’d like to make it a little more fancy than that. In comes InDesign.

Screenshot of the online statistics prototype I was working on

I don’t have a copy of InDesign server, and I’d love some training in scripting and InDesign server apps, but for the moment I’m working on a scenario with the InDesign CS4 client – here’s what I want to do.

I want a button on the prototype site that says “send me these stats”.. from the user’s perspective, they click it and next thing, an email shows up in their inbox with an attached PDF of beautifully formatted statistics, with the institution’s logo and so on. Sound nice?

From my perspective that means: I’d need the XSL for InDesign to transform the XML used by Fusion Charts and the prototypes XSL (in table view) – which would allow InDesign to import the data as an actual table, already mapped with table and cell styles. Obviously to automate all this and generate/email a final PDF I’d need the InDesign server, but hey, I’m just trying to prove a concept here that it’s doable.

It’s quite easy to import XML in to InDesign and have it all mapped with styles and automatic formatting. Take my XML for example:

<chart caption='Student Load (EFTSL) by Gender' subcaption='2006 Full Year' xaxisname='Student Modes' yaxisname='Student Load' palette='1'>
	<categories>
		<category label='Commencing' />
		<category label='Continuing' />
		<category label='All' />
	</categories>
	<dataset SeriesName='Male'>
		<set value='4275' />
		<set value='6245' />
		<set value='10520' />
	</dataset>
	<dataset SeriesName='Female'>
		<set value='3401' />
		<set value='4532' />
		<set value='7933' />
	</dataset>
	<dataset SeriesName='Both'>
		<set value='7676' />
		<set value='10777' />
		<set value='18453' />
	</dataset>
</chart>

And look at how I’ve done the XSL for InDesign, note the name space defs, in particular the aid5 one which is for the latest version of InDesign only ; )

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:aid="http://ns.adobe.com/AdobeInDesign/4.0/" xmlns:aid5="http://ns.adobe.com/AdobeInDesign/5.0/">
<xsl:template match="/">
<xsl:for-each select="chart">
<Root>
<Story>
<h3 aid:pstyle="h3"><xsl:value-of select="@caption"></xsl:value-of></h3>
<h4 aid:pstyle="h4"><xsl:value-of select="@subcaption"></xsl:value-of></h4>

<Table aid:table="table" aid:trows="4" aid:tcols="4" aid5:tablestyle="DataTable">

	<ColHeader aid5:cellstyle="BlankCell" aid:table="cell" aid:theader="" aid:crows="1" aid:ccols="1" > </ColHeader>

	<xsl:for-each select="categories/category">
		<ColHeader aid5:cellstyle="ColHeader" aid:table="cell" aid:theader="" aid:crows="1" aid:ccols="1" aid:pstyle="TableHeading"><xsl:value-of select="@label"></xsl:value-of></ColHeader>
	</xsl:for-each>

	<xsl:for-each select="dataset">

	<RowHeader aid5:cellstyle="RowHeader" aid:table="cell" aid:crows="1" aid:ccols="1" aid:pstyle="RowHeading">
		<xsl:value-of select="@SeriesName"></xsl:value-of>
	</RowHeader>

	<xsl:for-each select="set">    	
		<Cell aid5:cellstyle="TableBody" aid:table="cell" aid:crows="1" aid:ccols="1" aid:pstyle="TableText">
			<xsl:value-of select="@value"></xsl:value-of>
		</Cell>
	</xsl:for-each>

	</xsl:for-each>

</Table>
</Story>
</Root>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>

To get it in to InDesign, it’s quick as, you just import the XML using the above XSL file, easy peasy, and because the XSL transforms the XML in to a format InDesign ‘likes’, the table will be created properly.

Problems are always involved though. So far I haven’t solved these:

  • I had to set the table’s cols and rows manually in the XSL – this is totally not good. You can do all sorts of tricky things with XSL 2.0, so, I’m sure I could put the count of the desired nodes in to a variable and base the column and row count on that.
  • Look at how it comes in to InDesign – the table width!!! Or should I say LACK of table width. I want it to full width the table but I can’t set any static dimensions because the number of columns in tables will always be variable… what to do I’m not quite sure. I could probably create an InDesign script which makes the table the width of the page (within the margins) then distributes the columns evenly? Then again, I could also do some trickery and math in the XSL – count the columns, divide the page width by the number of columns and use the standard cell width in XSL?

Still, it does look promising nonetheless. I’ve seen lots of posts on the net with people having problems getting table data in to InDesign at all, so hopefully the above code snippets and stuff will help some people.

I’ll keep working on this though, and at some point, I’d LOVE to get my mittens on a copy of InDesign server to see what it can do to help me.





The stats of success

23 02 2010

Some time ago I posted about the eStudyGuide website I set up to match our InDesign study guide process. To explain it again, briefly, the XML from our InDesign TOCs get’s uploaded to a webserver along with the PDFs – the webpage reads the XML files and renders a download page for the course you’re looking for.

Well!  Anyway!  Term has started finally and this is the first semester that I’ve had the pages also hooked in to Google Analytics. I had this weird moment of doubt at some point, that the eStudyGuides weren’t really being used, and that we’d not really contributed anything of use to the University.  Not sure why on earth I thought that.  The stats for the first couple of weeks of Term prove how useful it is.

  • Over 4000 page views already
  • Over 1300 hits already
  • Over 5.5 minutes average on each page


eStudyGuide stats

I love Google Analytics, it’s damn awesome.  I’m looking forward to seeing the trends from now on, how each term differs from the next.

Overall, the InDesign XML features have really proved essential in all of this.  There are some glitches though that I believe I’ve never mentioned.  Some times carriage returns get tagged in the TOC when you map styles to tags – which means you get an empty XML tag in your XML document.

You’ll need error checking to make sure you’re not rendering that empty tag as anything on your website, especially if the count of XML tags matches a document count like the eStudyGuide set up.

I wonder though, would it be worth doing an XML Schema and having InDesign validate it’s output with the schema when we export our TOC XML?

The whole process, exporting the XML and PDFs, and then uploading the files is a bit labourious too, and mistakes happen.  With a bit of perl/php mastery, one of the more technical sauvy guys in our team has done some brilliant reporting around the setup, as well as a script that handles that uploading – a must have.  It checks document counts with the XML tags to make sure they match, checks for typos even, missing files, and more.  Many thanks to Damo for his help.

Onwards and upwards, I wonder what the next lot of improvements to the eSGs could be 🙂





Putting statistics online

4 02 2010

We have a statistics website at work with an enormous amount of statistics available for PDF download.  I like PDF downloads.  But.  Just to get a table of statistics, a user has to click through 5 layers of navigation!  The menu, sub menu, sub sub menu, sub sub sub menu…. you see?  It’s not really ideal – all that just to get a downloadable PDF, which usually opens up in an annoying new window as well.

The sub menu navigation on the site is basically like choosing a statistics category, and graph metrics.  So, I figure it’s got to be cleaner if we move to some sort of online graphing tool, right?  We could have a single page even, where a form (or cascading forms) refresh the graph and it’s metrics as you click.  All in one spot, tidy and neat, and fun.  It’d also be easy to have a tab for table-view for people who don’t like graphs.

There’s a lot of problems to overcome before I can start though.  I wrote a report for my boss, just to itemise what we’d need to consider, to move to online graphing.  The stuff below is a part of the report.  If anyone out there has any wonderful ideas about the best way to go, please let me know.  I’m mostly interested in how to get the data in to XML (from excel) in an automated way; and how to conceptualise heavily layered statistics in to graphical break downs.  Is all this a good idea?

As a start I actually did a tiny prototype using the free tool SWF/XML Charts.  It’s a good tool, easy to use, looks reasonable to the eyes – and most of all it was easy to flex in to something more powerful, with a bit of php, ajax and jquery.  I had it set up to work really well – the forms refreshed the graph on the fly; the table view read the same XML file as the Flash graphing tool, and transformed it in to a HTML table (in the table tab).  It was so dynamic and I loved it as a basic prototype – but I’m just not sure of the best next step yet.

Example graphing prototype

It worked really well - there was one tricky part though, the tabs loaded content with jquery ajax and if you changed the forms whilst in table view (for example) the graph view didn't refresh because it didn't exist. Luckily it was fairly straight forward to fix with some callback methods in the ajax call.

Oh, and another tip – the tool hated having symbols parsed in the URL for the xml data source. So my ajax table loading might be like this:  (all these variables are actually values from the form, and the events are triggered by changing the form)
$("#Table").load("tableContent.php?data=TrendData_MainIntake" + "&graph=" + graph + "&category=" + category);

But the url variable in the script that loaded the flash graphing tool had to be like this, using hex values in the path:
url = "graphContent.php%3Fdata%3DTrendData_MainIntake" + "%26graph%3D" + graph + "%26category%3D" + category + "%26type%3D" + type;

It’s fun getting all this stuff to work. I hope it turns in to an excellent project, a small part of my report is as follows if you’re interested.
Read the rest of this entry »





How to serve up print-documents on the web?

23 11 2009

In a quest for continuous improvement, we’re having a think about our study guides. Most authors have their documents online for students, be it PDFs (mostly) or word docs.. what the students do with those documents, I’m not entirely sure, and as far as I know, there’s no stats or research that might answer that question.

Even the stats for the eStudyGuide pages (discussed in earlier post – taking the xml from indesign to make a web front end for the study guides) don’t indicate any trend, the stats only show that people are looking at the front end, from where, for how long etc. Actually, there’s a possible answer there, I wonder if there’s any server stats on the PDFs, would the stats indicate how long the document is open in the browser, or would the stats just record the hit? I’ll have to find out. There’s got to be some data we can look at to know more about usage.

In the mean time, tell me what you would do with a PDF study guide if you were studying:

So, anyway, back to the point of this, what is really the best way to serve up the documents online? If students are indeed reading on screen, perhaps we should put some time into looking at a better way to present it all, Adobe Digital Editions for instance might be a winner. Like Acrobat Reader, its a free program, but it’s for managing and viewing eBooks. Students would be able to manage and read all their study guides on their computer (making them more convenient, mobile, and so on), plus, the epub format would be easy as to spit out of InDesign (what our study guides are created in).





Incrementing and logic in XSL

17 11 2009

I’ve been learning XSL for the design of the learning guide I mentioned in my last post. I just had a bit of a win with incrementing in the XSL! I searched the net far and wide and didn’t find anything that suited what I wanted in terms of incrementing, most things didn’t even work anyway.

The XML I’m transforming describes the content in terms of “sections”. The XSL transforms those sections into a jquery tabbed interface. So basically, I need the title attributes of the tab links, to match the id attributes of the div containers. I also need these to be unique for the jquery tabs to work. Thus, I wanted to increment the values.

Here’s a sample of the XML, just a simple example – an item is a module in the learning guide, it contains various information, part of which includes the actual ‘content’ for that module:

<itemContent>
<section title="Start">
<sectionContent>first section of this module</sectionContent>
</section>
<section title="Middle one">
<sectionContent>content for middle section</sectionContent>
</section>
<section title="Finish">
<sectionContent>content for third section here</sectionContent>
</section>
</itemContent>

Then, the XSL that transforms and increments. I had it all around the wrong way initially, I was trying to use variables and addition, loops etc, to do my incrementing.. but really, all I had to do was identify the node position and use that as a parameter in my for-each loop!

<div id="tabs">
	<xsl:for-each select="itemContent/section">                    
		<xsl:param name="i"><xsl:number value="position()" /></xsl:param>
		<a href="#" title="Section_{$i}"><xsl:value-of select="@title" /></a>              		
	</xsl:for-each>
	<div class="clear"></div>
</div>

<xsl:for-each select="itemContent/section">
	<xsl:param name="i"><xsl:number value="position()" /></xsl:param>
	<div id="Section_{$i}" class="hiddencontent">
		<h3><xsl:value-of select="@title" /></h3>
		<xsl:copy-of select="sectionContent" />
	</div>
</xsl:for-each>

It works a charm and it renders like this in the browser:

HTML render of the XSL incrementing

Note the incrementing and matching title/id attributes..





Media-rich Learning Guide

6 11 2009

We had this little problem last year – we really wanted SOMETHING to do an online study/learning guide with.. well not just ‘something’, we wanted it to be media rich in that it is not just visually appealing but allowed all sorts of GUI features (like popups, tabs etc) so that it could have more media-rich content.

The reason at the time was because the course we were working on had SO many resources, from all over the place, and we thought it might be a good thing to bring everything together, in to context.  (and to make it look nice).

I started thinking of ideas for a GUI and realised one of my favourite interfaces would have to be iTunes!  The interface is called cover flow and it’s visually delicious to say the least.

iTunes coverflow

This is the iTunes coverflow GUI, nice hey?

As it turned out, after some searching the net for Flash resources, I found this awesome Open Source Initiative Flash-based cover flow project.  The beauty of it, like the beauty of iTunes, is it basically gets all its data from an XML file.  So that got me thinking.

What if the learning guide was entirely in XML?  Well, why not.  You could request a course code in your web browser, a php page could parse your request to Flash, which then loads the XML learning guide menu in coverflow, and voila, ajax the content in to the page with each click in the flash cover-flow!

I modified the Flash-based coverflow to suite our purpose.  I made the initial course trial static, in that the content was being loaded from html files, and the course-specific XML only contained details on the course topics and titles etc.  Now however I’m working on the full deal, the entire learning guide in XML.  It’s all coming together rather nicely.

HLG

We've nick-named it the HLG (hypertextual learning guide). You select your study module from the coverflow menu, and all the content loads below via ajax. The content sits in the course XML, and the XSL handles the rendering of all the variations in that content. This example has some readings, but there could also be videos, podcasts, all sorts of stuff.

Learning XSL has been a highlight of the last week, I’m still quite a noob at it, but so far I’ve done some awesome things – I can handle all sorts of variations in the XML (which means variations in content) and render it on the page nicely – videos come up with a video icon and use the jQuery prettyBox plugin, documents come up with a document icon, abstracts have the abstract viewable via a jQuery clueTip plugin.. its all coming together.

The setup

I have some more work to do with the XML/XSL, but I think it will all go well. At the end of it, I'll basically have a single php file, that takes course code requests.. the page will talk to a library of standard resources like the images in the cover-flow menu, the background images, css, and javascripts. It'll then talk to the requested XML (via a php transform and XSL file) to get all the details and content.

Phase 3 will be doing an administration interface, to manage the XML.. ohh fun.





Web front end for InDesign docs

5 11 2009

Well well.. I’ve been out of touch with the InDesign stuff I was really getting in to last year.  Other projects, as well as the restructure and fallout from that.  I did however pick things up a little just recently, and finished making a first draft front end to display the Study Guides from InDesign.

As you may/may not know, part of our InDesign process with the study guides allows us to export XML from the Table Of Contents (TOC).  This is a great spot to get some really usable data about the document, for instance, how many chapters and what are the chapter titles!

After creating the PDFs, and exporting the XML from the TOC, we simply dump it all on a web server.  I made a nice little PHP page that takes requests and displays an interface for students to download those PDFs – the PHP reads the requested course’s XML file, creates the list with the chapter titles and so on.

Why’d I bother doing this?  A couple of reasons really… Well okay a few:

  • Its fun.
  • It saves academics time by setting up a page for them where students can download the chapterised PDFs for their Study Guide.
  • It also saves the academic the time of splitting their whole document PDFs up in to chapter PDFs, because we do it.

It works well for us really, I mean, all we do is export the goods and throw it on a server, it couldn’t be more simple.  The outcome is really quite nice.  Next challenge will be how to integrate this more with the new LMS, Moodle.

e Study Guide Page

This is the end result, the web front end for the InDesign XML & PDFs

In terms of the web front end, it was quite easy with PHP to read the XML.  I didn’t need to do anything fancy at all, I suppose it is pretty simple XML.  In fact, I used the simpleXML php module.

By parsing in the courseID as a variable in the URL, the page loads the requested XML file

//capture courseID from URL
$courseID=$_GET["courseID"];

//load the xml that contains the info on the documents
$pathExt = ".xml";
$docInfoPath = $courseID . "/eStudyGuide/" . $courseID . $pathExt;
$docInfoXML = simplexml_load_file($docInfoPath);
$docTitles = array();

Then looping through to display the docs was easy as this!

$i = 1;

foreach ($docInfoXML->TOClev1 as $docTitles) {
	if (strlen($docTitles)!=3) {
		echo "<li>
			<a href='" . $courseID . "/eStudyGuide/" . $courseID . "_" . $i . ".pdf'>" . str_replace('    ', ":  pg", $docTitles) . "</a></li>";                            
		$i++;
	};
 };