Pulling a Report using the Reporting API

This week we want to show an example of how to pull a report out of Adobe Analytics using the Reporting API. We’ll start with a simple report, then add metrics, a subrelation and a filter in the next post. We’ll also apply a segment.

The Report

Most Reporting API projects do not start with a full brief. Your friendly marketer will likely walk up to your desk or send you an email stating “we need an automated way to pull these campaign reports. Adobe say you can use the Reporting API. Make it so.”

Our advice is to stop right here.

Bad or incomplete specifications are an issue at the best of times. In this case add a cultural rift, misunderstandings of capabilities plus a distinct hunch that the marketer himself might not know what exactly it is he wants automated.

So walk over to his desk and sit down. Quiz him. Don’t stop until you know that he does actually want you to programmatically pull data and what he wants to do with it.

Then, and only then, have him log into SiteCatalyst and show you the actual report he wants. Or, if he is so inclined, have him show you in Report Builder.

The Reporting API

The Reporting API gives you access to the aggregated reports that SiteCatalyst and Report Builder show as well. There is no hidden trick, it just pulls the exact same data. If you run your report against the Reporting API at a specific time and your marketer pulls his at the same time, your numbers will match exactly.

The report in SiteCatalyst or Report Builder is therefore a really good prototype that’ll help you a) define your report and b) debug really easily.

Let’s take this Campaign report as an example:

[Screenshot]

Campaign Report in SiteCatalyst

You can see that the dimensions are tracking codes and the metric is “Visits”, easy.

So let’s build it!

We’ll use Java for the examples, mostly because it translates fairly easily to a .NET language, but also because accessing the APIs using PHP is easier. So if you’re using PHP, you should not have too much trouble translating the code examples below.

We’ll use SOAP this time round. A REST-ful example will follow.

Setup

You’ll need a couple of libraries. Versions change and technologies move on, of course. At the time of this writing, we are using the following JARs:

  • axis
  • wsdl4j
  • jaxrpc
  • saaj
  • xercesImpl
  • wss4j
  • axis-ant
  • xmlsec
  • commons-codec
  • commons-discovery
  • commons-httpclient
  • serializer
  • xalan
  • resolver

You will also need the WSDL file that describes the Reporting API interface. Go and download it from the SiteCatalyst UI. You can find it under Admin > Admin Console > Company and then behind the Web Services link.

[Screenshot]

WSDL Download in SiteCatalyst

Note the two check boxes at the bottom? I’d check them to avoid cases where a weird character in the stream breaks the results.

Authentication

You can find your username and password (called “secret”) in the SiteCatalyst UI under Admin > Admin Console > User Management, then click Edit Users, find yourself and click your login. Or your friendly marketer can find it for you.

[Screenshot]

Username & Secret for the API

The Reporting API uses “UsernameToken” authentication, not the most easy model I have seen.

You need to instruct axis to use it by generating a client-side descriptor file, usually named client_deploy.wsdd, like so:

  <?xml version="1.0" encoding="UTF-8"?>
  <deployment xmlns="http://xml.apache.org/axis/wsdd/" xmlns:java="http://xml.apache.org/axis/wsdd/providers/java">
    <transport name="http" pivot="java:org.apache.axis.transport.http.HTTPSender"/>
    <globalConfiguration>
      <requestFlow>
        <handler type="java:org.apache.ws.axis.security.WSDoAllSender">
          <parameter name="action" value="UsernameToken" />
          <parameter name="user" value="USERNAME" />
          <parameter name="passwordCallbackClass" value="YOUR.FQDN.HERE.PWCallback" />
          <parameter name="addUTElements" value="Nonce Created" />
          <parameter name="mustUnderstand" value="true" />
          <parameter name="passwordType" value="PasswordDigest" />
        </handler>
      </requestFlow>
    </globalConfiguration>
  </deployment>

For axis to use this file, pass it to your JVM as a paramter: -Daxis.ClientConfigFile=client_deploy.wsdd

You also must provide a callback class and method like this:

  @Override
  public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
    for (int i = 0; i < callbacks.length; i++) {
      if (callbacks[i] instanceof WSPasswordCallback) {
        WSPasswordCallback pc = (WSPasswordCallback) callbacks[i];
        // set the password given a user name
        if (AxisTest.USERNAME.equals(pc.getIdentifer())) {
          pc.setPassword(generateAuthKey());
        }
      } else {
        throw new UnsupportedCallbackException(callbacks[i], "Unrecognized Callback");
      }
    }
  }

The generateAuthKey() method simply passes back the password.

In order to test whether your authentication works, I’d suggest you call the Company.GetTokenCount method, which returns the number of tokens you have left for the month.

If that works, move on to the next step – building the report.

Report Definition

This is the most important part: you need to tell the Reporting API which report you want.

With the report you got from your friendly marketer, this step is relatively straight-forward. All we need to do is instantiate an object of the reportDescription class and populate it’s members accordingly.

  // Create a report description
  ReportDescription reportDesc = new ReportDescription();
  reportDesc.setReportSuiteID("<RSID>");
  ReportDefinitionLocale locale = ReportDefinitionLocale.en_US;
  reportDesc.setLocale(locale);
  reportDesc.setDateTo("2013/04/01");
  reportDesc.setDateFrom("2013/04/30");
  ReportDefinitionMetric[] metrics = new ReportDefinitionMetric[1];
  metrics[0] = new ReportDefinitionMetric();
  metrics[0].setId("visits");
  reportDesc.setMetrics(metrics);
  ReportDefinitionElement[] elements = new ReportDefinitionElement[1];
  elements[0] = new ReportDefinitionElement();
  elements[0].setId("tracking_code");
  reportDesc.setElements(elements);

The setMetrics method allows you to specify the metric(s) you need while the setElements method is used to specify the dimension(s).

Make sure you use the actual report suite ID instead of “RSID” above. Your friendly marketer should be able to tell you, or you can find it yourself.

“Run” the Report

Now all you need to do is to “queue” that report so the system can run it.

The Reporting API has three methods for queueing: Report.QueueRanked, Report.QueueTrended and Report.QueueOvertime. They do different things.

Report.QueueRanked creates ranked reports like the one in the screenshot above, basically listing dimensions in descending order of whatever the metric is. A “most popular pages” report is a good example.

Report.QueueTrended creates reports that show you how up to five dimensions develop over time for a specific metric. An example would be a report that shows how marketing channels evolve.

[Screenshot]

Trended Report

Report.QueueOvertime creates a simple overview report of multiple metrics, similar to the “Key Metrics” report in SiteCatalyst.

[Screenshot]

Overtime Report

Note: the different queue methods will reject a report description that violates their constraints.

  • A ranked report can have up to 9 metrics and up to two dimensions. Those must either both be traffic variables or conversion variables, but not a mix of both.
  • A trended report can accept one single metric.
  • An overtime report takes no dimension at all but multiple metrics.

Next you wait for the processing to be done. It is impossible to predict in general how long it’ll take, so you can either run a couple of tests, or just have your program go into a sleep loop and check the status of the report using Report.GetStatus every second or so.

You should look for either “done” or “failed” as a response. We think it’s obvious what those mean.

Retrieve the Data

When the waiting is over, you can retrieve the resulting data using the Report.GetReport method. You’ll get back a reportResponse object which carries status as well as the data in a nested report object. Again, you should check that the status is “done”.

Now is where the printout from your friendly marketer really is needed: the data structure that you get back will be a bit complex, depending on how complex your reportDefinition is, of course. Use the original report to figure out where in the structure you can find the data that you need.

As an example, a result for the report definition above might look like this:

[Screenshot]

Report vs API Results

Next time we’ll extend the reportDescription to make it more interesting.

And that’s pretty much it. Happy coding!

About

German expat living in Switzerland (formerly UK and France). Consultant and member of the Multi-Solutions Group at Adobe, working with the Digital Marketing Suite. Father of 4 girls.

Posted in Automation, Principles
17 comments on “Pulling a Report using the Reporting API
  1. saroshelahi2013 says:

    Hi Jan,

    Thanks for sharing and the level of detial it is great. Can you please shed some light on token usage when using the reporting API? How many per report are used if that is how they are consumed. Also is this the best way to extract data from Adobe Analytics in order for it to be displayed on plasma screens for general consumption?

    Like

    • jexner says:

      Tokens: I did not mention them because since May 23rd they don’t matter anymore. Access to the Reporting API is now free. It might be throttled, but it doesn’t cost money anymore. The current guidance from Engineering is to run small requests multiple times rather than single big ones.

      Plasma screens: yes, this is perfect for building custom screens or dashboards. Best uses cases that we’ve seen: C-level office dashboard, and information screen at reception.

      Like

  2. […] you are using the Reporting API, your endpoint depends on the data centre (“api” points to San Jose, “api2″ […]

    Like

  3. […] you know that since the release on June 20th, you can get data out of the Reporting API with seconds of […]

    Like

  4. […] we’re looking at the Reporting API and little-known features, let’s use the new anomaly detection that was introduced in […]

    Like

  5. […] have “mentioned” the Reporting API before, a means of getting information out of Adobe analytics […]

    Like

  6. […] is one more thing that we need to mention, one more thing you can do to help her: you can use the Reporting API to pull data for her, and you can make fancy displays or dashboards for her. The data that you pull […]

    Like

  7. […] the access control via tokens system was dropped some time ago, those of us who had used the Reporting API had to change how we used […]

    Like

  8. […] If you think about the processing of hits, you can see that there are multiple points that tests could hook into. The simpler tests could look at (or live in) the doPlugins method in the s_code.js file, whereas more involved scenarios would have tests that looked at the processed data, maybe using the Reporting API. […]

    Like

  9. […] build upon the previous articles on the subject and start with the […]

    Like

  10. Tony Fraser says:

    Thank you SO MUCH for this Jan.! This article is so extremely useful I’d say you should sell it to Adobe. I got myself up and running end to end in eclipse in just a couple of hours, and I started without axis –never worked with it before. The only thing that would make this article perfect is letting people know what kind of a callback this is. Is it an Axis callback? Posting the full java class here would be so useful.!

    @Override
    public void handle(Callback[] callbacks) throws IOException,

    Like

  11. slowbalboa says:

    Ok, got that figured out. Here it is for all your other readers Jan.

    For anybody else reading, I went through quite a bit to reproduce Jan’s work. I learned a lot. This is an older blog entry though, so let me throw in a couple of updates. First and most importantly, this: java:org.apache.ws.axis.security.WSDoAllSender is available in the version of wss4j that Jan shows in his screen shot. I couldn’t get anything working in any newer versions, I say stick with that one unless you know exactly how Axis works.

    Here’s my version of the callback code.

    import javax.security.auth.callback.Callback;
    import javax.security.auth.callback.CallbackHandler;
    import org.apache.ws.security.*;
    import java.io.IOException;
    import javax.security.auth.callback.UnsupportedCallbackException;

    public class TjCallback implements CallbackHandler {
    public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {

    for (int i = 0; i < callbacks.length; i++) {
    if (callbacks[i] instanceof WSPasswordCallback) {
    WSPasswordCallback pc = (WSPasswordCallback) callbacks[i];
    pc.setPassword("your password");
    System.out.println("pause");
    } else {
    throw new UnsupportedCallbackException(callbacks[i], "Unrecognized Callback");
    }
    }
    }

    }

    Also, I couldn't get the wsdd thing working right. I saw another thread that used a different invocation using the wsdd file as an engine configuration. That seemed to work for me.

    try {
    EngineConfiguration config = new FileProvider("/Users/xyz/Documents/workspace/SiteCatalyst/clientdeploy.wsdd");
    AdobeAnalyticsServiceBindingStub stub = (AdobeAnalyticsServiceBindingStub) new AdobeAnalyticsServiceLocator(config).getAdobeAnalyticsServicePort();
    stub.setPassword("secret:secret");
    Report_queue_item[] items = stub.reportGetQueue();
    System.out.println("hello world");

    } catch (ServiceException | RemoteException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }

    Final note:

    Jan, much respect for figuring all this out. And many many thanks for posting. I don't think I could have done it in less than a month, instead it took me two days. I'm pretty good with Hadoop and HDFS and such, and a 3rd party called Domo. LMK if you ever need to do any brainstorming. Find me here: tonyandjaime.com

    tony

    Liked by 1 person

  12. slowbalboa says:

    Hey Jan. Man this is so crazy. These seemed to work fine…

    (this)
    Report_queue_item[] items = stub.reportGetQueue();

    (and)
    Report_suite_props[] rsProps = stub.reportSuiteGetProps(rsid);
    Prop[] props = rsProps[0].getProps();
    for (int i = 0; i < props.length; i++){
    System.out.println("Properties: ID=" + props[i].getId() + "\tName=" + props[i].getName());
    }

    But when I try to run.. (which is almost the same thing)
    Report_suite_events[] rsEvents = stub.reportSuiteGetEvents(rsid);

    Event[] events = rsEvents[0].getEvents();
    for (int i = 0; i Events: ID=” + events[i].getId() + “\tName=” + events[i].getName());
    }

    I get a deserializer error…

    About to throw this 0
    Feb 16, 2016 3:59:23 PM org.apache.axis.client.Call invoke
    SEVERE: Exception:
    org.xml.sax.SAXException
    java.lang.IllegalArgumentException
    at org.apache.axis.encoding.ser.SimpleDeserializer.onEndElement(SimpleDeserializer.java:176)
    at org.apache.axis.encoding.DeserializerImpl.endElement(DeserializerImpl.java:502)
    at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
    at org.apache.axis.message.SAX2EventRecorder.replay(SAX2EventRecorder.java:171)
    at org.apache.axis.message.MessageElement.publishToHandler(MessageElement.java:1141)
    at org.apache.axis.message.RPCElement.deserialize(RPCElement.java:236)
    at org.apache.axis.message.RPCElement.getParams(RPCElement.java:384)
    at org.apache.axis.client.Call.invoke(Call.java:2467)
    at org.apache.axis.client.Call.invoke(Call.java:2366)
    at org.apache.axis.client.Call.invoke(Call.java:1812)
    at com.adobe.marketing.analytics.AdobeAnalyticsServiceBindingStub.reportSuiteGetEvents(AdobeAnalyticsServiceBindingStub.java:6925)
    at com.tj.Main.main(Main.java:58)
    Caused by: java.lang.IllegalArgumentException
    at com.adobe.marketing.analytics.Event_visibility_enum.fromValue(Event_visibility_enum.java:31)
    at com.adobe.marketing.analytics.Event_visibility_enum.fromString(Event_visibility_enum.java:36)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.axis.encoding.ser.EnumDeserializer.makeValue(EnumDeserializer.java:53)
    at org.apache.axis.encoding.ser.SimpleDeserializer.onEndElement(SimpleDeserializer.java:172)
    … 11 more

    About to throw this 6!
    AxisFault
    faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
    faultSubcode:
    faultString: org.xml.sax.SAXException
    java.lang.IllegalArgumentException
    faultActor:
    faultNode:
    faultDetail:
    {http://xml.apache.org/axis/}stackTrace:org.xml.sax.SAXException
    java.lang.IllegalArgumentException
    at org.apache.axis.encoding.ser.SimpleDeserializer.onEndElement(SimpleDeserializer.java:176)
    at org.apache.axis.encoding.DeserializerImpl.endElement(DeserializerImpl.java:502)
    at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
    at org.apache.axis.message.SAX2EventRecorder.replay(SAX2EventRecorder.java:171)
    at org.apache.axis.message.MessageElement.publishToHandler(MessageElement.java:1141)
    at org.apache.axis.message.RPCElement.deserialize(RPCElement.java:236)
    at org.apache.axis.message.RPCElement.getParams(RPCElement.java:384)
    at org.apache.axis.client.Call.invoke(Call.java:2467)
    at org.apache.axis.client.Call.invoke(Call.java:2366)
    at org.apache.axis.client.Call.invoke(Call.java:1812)
    at com.adobe.marketing.analytics.AdobeAnalyticsServiceBindingStub.reportSuiteGetEvents(AdobeAnalyticsServiceBindingStub.java:6925)
    at com.tj.Main.main(Main.java:58)
    Caused by: java.lang.IllegalArgumentException
    at com.adobe.marketing.analytics.Event_visibility_enum.fromValue(Event_visibility_enum.java:31)
    at com.adobe.marketing.analytics.Event_visibility_enum.fromString(Event_visibility_enum.java:36)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.axis.encoding.ser.EnumDeserializer.makeValue(EnumDeserializer.java:53)
    at org.apache.axis.encoding.ser.SimpleDeserializer.onEndElement(SimpleDeserializer.java:172)
    … 11 more

    {http://xml.apache.org/axis/}hostname:Atom

    I modified the code to say this to know where it was failing. Pretty clear there is something wrong with my Axis somewhere.

    try {
    System.out.println(“About to throw this 0”);
    java.lang.Object _resp = _call.invoke(new java.lang.Object[] {rsid_list});
    System.out.println(“About to throw this 1”);
    if (_resp instanceof java.rmi.RemoteException) {
    System.out.println(“About to throw this 2 !”);
    throw (java.rmi.RemoteException)_resp;
    }
    else {
    extractAttachments(_call);
    System.out.println(“About to throw this 3 !”);
    try {
    System.out.println(“About to throw this 4 !”);
    return (com.adobe.marketing.analytics.Report_suite_events[]) _resp;
    } catch (java.lang.Exception _exception) {
    System.out.println(“About to throw this 5 !”);
    return (com.adobe.marketing.analytics.Report_suite_events[]) org.apache.axis.utils.JavaUtils.convert(_resp, com.adobe.marketing.analytics.Report_suite_events[].class);
    }
    }
    } catch (org.apache.axis.AxisFault axisFaultException) {
    System.out.println(“About to throw this 6!”);
    throw axisFaultException;
    }

    And when I run a report,

    ReportDescription desc = new ReportDescription();
    //ReportDefinitionLocale locale = ReportDefinitionLocale.en_US;
    desc.setReportSuiteID(“myreportsuite”);
    desc.setDateTo(“01-15-2016”);
    desc.setDateTo(“02-15-2016”);
    ReportDescriptionMetric[] metrics = new ReportDescriptionMetric[1];
    metrics[0] = new ReportDescriptionMetric();
    metrics[0].setId(“40”);
    ReportQueueResponse queResponse = stub.reportQueue(desc);
    System.out.println(“Report Queue: ” + queResponse.getReportID());

    I get…

    faultDetail:
    {http://xml.apache.org/axis/}stackTrace:org.w3c.dom.DOMException: WRONG_DOCUMENT_ERR: A node is used in a different document than the one that created it.
    at org.apache.xerces.dom.ParentNode.internalInsertBefore(Unknown Source)
    at org.apache.xerces.dom.ParentNode.insertBefore(Unknown Source)
    at org.apache.xerces.dom.NodeImpl.appendChild(Unknown Source)
    at org.apache.axis.message.SOAPFaultBuilder.onEndChild(SOAPFaultBuilder.java:305)
    at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1090)

    Seriously feels like an Axis thing.

    I’ve been banging my head against the wall. Tried migrating to axis 2, but they don’t support rpc style declarations. Went back to 1.5.1, then back to 1.5.3 for both wsdl4j and wss4j. Must have built those classes and checked my classpath in eclipse a dozen times.

    And what makes it worse is the eclipse TCP Monitor won’t work because this is all on https dev. Figured at least there was some XML coming back that wasn’t right that I could verify.

    Any suggestions? Also, there used to be a method for getMetrics, what happened to it? I can’t seem to find all my metrics.

    Tony Fraser from NYC.

    Like

    • Jan Exner says:

      Hi Tony,

      I’m afraid I ditched AXIS and SOAP the moment I learned how to access the Reporting API in a RESTful way. Haven’t touched the old code since then.

      Can you switch to REST at all? Might be easier…

      Like

  13. […] the past, I have written about first steps in Java, so I shall not repeat myself […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 1,398 other followers

%d bloggers like this: