Features Archives - Page 2 of 4 - MiaProva

Category: Features Page 2 of 4

Adobe Target Parent and Child Activities

WIth Adobe Target, you can have two Activities work together to assign users to groups after some behavior has occurred or any other event that you would like. This video outlines this very helpful approach.

Video Transcript

Live Dashboard Q&A

We received some great questions from a video we made about the MiaProva Live Dashboard. This video is a response to those questions.

Video Transcript

MiaProva Live Dashboard

MiaProva’s Live Dashboard provides organizations with various tools to manage any Activities in flight. This quick 2-minute video gives you an overview of some of the functionality available here.

Video Transcript

MiaProva Product Updates August 22, 2022

MiaProva regularly makes product releases once or twice a month on average. This video outlines some of the updates made this week.

Monday, August 22, 2022 Product Release

– UI updates to the Idea Lab
– Adding Adobe Target Audiences and Adobe Target Offers to Program fields
– Bug Fix: Removed Alerts sent via email from disabled Report Suites
– Various architectural and performance enhancements
– New Filter on Awaiting Summary
– Updates to naming conventions under Notifications

As always, please reach out with any questions and, Customers, if there are features that would be helpful, please let us know at support@miaprova.com

Video Transcript

Amplitude and Adobe Target

Amplitude is a Digital Optimization System that we have been asked about quite a bit lately. It turns out, MiaProva has some friends there from our Omniture and Adobe days, and so we got a much closer look at the solution and some of the use cases that organizations are using it for.

They offer a free trial that allows you to send 10,000,000 hits per month, and so we decided to kick the tires and see if we could create an integration for those organizations that use Amplitude and Adobe Target.

Image from: https://amplitude.com/digital-optimization-system

Amplitude Data Consumption

We are by no means experts and have minimal experience with Amplitude, but from what we figured out so far, there are many different ways to collect data via SDK Libraries. Given that most organizations using Adobe Target leverages Javascript, we decided to create our Adobe Target custom integration based on their Javascript SDK.

Amplitude SDKs

Users and Events

Greatly simplifying things, with Amplitude, you send in a Visitors ID (or use theirs), pass in any Properties related to that ID, and any Events as you wish.

Here is an elementary example of passing to Amplitude that a Visitor is a “Cubs” fan:

var userProperties = {
    favorite_baseball_team: "Cubs"
};
amplitude.getInstance().setUserProperties(userProperties);

This data would then be correlated with the User that was initially defined. An event represents something that has happened like ‘clicked,’ ‘searched,’ ‘hovered,’ ‘started a video,’ etc… Really anything that all and these Events can be Arrays and have Properties similar to how User data is passed. Here is what an Event call looks like:

var event = "MiaProva_trial";
amplitude.getInstance().logEvent(event);

We’ve configured this event to fire when a visitor arrives on our trial page: https://www.miaprova.com/trial as a straightforward example. This documentation goes into much more advanced approaches as well.

Passing in your Adobe Target data

Adobe Target has some wonderful mechanisms to make data available for other solutions like Amplitude. Those organizations that use our popular Chrome Extension get to see Adobe Target Response tokens as one way to make the Activity meta-data visible. We did something similar with our approach to Amplitude.

We first had to decide whether we wanted to push the data as an “Event” Property or “User” Property. This decision has some significant implications for the Analysis and the management of data. While folks can debate the approaches, the best approach is a “User Property.” Adobe Target defaults to the Visitor counting methodology, and for excellent reason. Participating in a test is not like clicking a button or visiting a page, but rather something tied to the User so that it can be correlated to Events after entering into the Activity.

With that in mind, this is the single step needed to push your Adobe Target test data to Amplitude. Keep in mind; these steps assume you have Amplitude snippet in place or the npm/yarn installation. Modifications can be made as needed depending on how you manage Adobe Target’s deployment – we are using Adobe Launch.

  1. You need to add this code to the bottom of your Adobe Target at.js file. This code can be added via the Adobe Target UI before your download the at.js file or it can be custom code added via Launch to execute after Target is Loaded.
     document.addEventListener(adobe.target.event.REQUEST_SUCCEEDED, function (e) {
         if (e.detail.responseTokens) {
             var rt = e.detail.responseTokens,
                 ids = [];
             window.targetExperienceList = '';
             for (var i = 0; i < rt.length; i++) {
                 var inList = false;
                 for (var j = 0; j < ids.length; j++) {
                     if (ids[j] == rt[i]['activity.id']) {
                         inList = true;
                         break;
                     }
                 }
                 if (!inList) {
                     ids.push(rt[i]['activity.id']);
                     targetExperienceList += (targetExperienceList ? ',' : '') + rt[i]['activity.name'] + ':' + rt[i]['experience.name'] + ':' + rt[i]['experience.trafficAllocationType'];
                 }
             }
         }
         if (window.targetExperienceList) {

             var values = [targetExperienceList];
             var identify = new amplitude.Identify().set('Activity:Experience', values);
             amplitude.getInstance().identify(identify);

         }

     });

Pretty easy, right!? Let us break things down so all can see what is going on with this code because developers can use this code to push data to any solution, including CDPs, data layers, etc…

Line 1: This is an event listener that will execute the code within it after a successful Adobe Target server call (mbox call:) has taken place.

Lines 2 – 19: We are creating a window scoped variable called “targetExperienceList” that will have a value that will equal a concatenation of all “Activities, Activity Experience, and Activity Allocation” that were tied to the response of the successful Adobe Target server call (mbox call:). On MiaProva.com, this is what the value looks like:

You won’t typically see this sort of thing, but we have quite a few Activities running here for demonstration purposes. For those of you not familiar with “Activity Allocation,” that value is only pertinent to Automated Personalization and Auto-Target, where Adobe has Control and Targeted Groups. All other Activity types will have ‘undefined’ as a value.

Lines 20 – 26: Here, we are first checking to see if there is a value for “targetExperienceList”. This value would exist if the Adobe Target server call (mbox call:) had Adobe Target Response Tokens that showed that the Visitor was in a live Activity. If so, then it would execute Lines 22-24, which pushes all the Adobe Target data to Amplitude as User Properties. This could differ from the above User Properties because we treat the values like an array if the Visitor gets into more than one Adobe Target Activity on a single Adobe Target server call (mbox call:).

And there you have it:

Then you can analyze as you wish, create user cohorts (not available in free trial), create dashboards, visualize paths, etc…

Happy testing, folks!

Why are we paying for Adobe Target Premium?

Optimization Management is so much more than managing the intake and output of tests. A true Optimization Management Platform enables the organization to monitor tests or groups of tests and answers the big questions about their testing program as a whole and their investments in resources and technologies.

MiaProva automatically stitches the organizational meta-data related to Activities to data from Adobe Target and Adobe Analytics. This approach allows organizations to use this data to analyze and manage their tests and what offline variables are impacting the program – similar to how organizations use eVars and list variables in Adobe Analytics.

One of the many ways organizations use this is to understand how Adobe Target Premium is adding value to their Optimization Programs. MiaProva automatically classifies Activities that are specific to Adobe Target Premium as a dimension for Program Analysis. Doing so exposes a potential problem that needs addressing if limited or no value is being realized or creates an opportunity to expand on any value that is being realized.

MiaProva just onboarded two organizations that have made significant investments with Adobe and are heavy users of Adobe Target. Both organizations have launched more than 100 Activities launched since January of 2021. While mapping test tickets or test design documents to Adobe Activities to put all that data to life, MiaProva was able to immediately highlight for both of these organizations that they were not receiving the amount of value they should be with Adobe Target Premium.

In this blog post, we are going to highlight how we were able to identify this opportunity (and others) for these two organizations, what is being done to change things, and a great example of another MiaProva client who sees considerable value in Premium and MiaProva has used the ROI models to expand on their use of the Automated Personalization.

Optimization Program Macro-Analysis

MiaProva automatically creates dimensions related to the Adobe data such as “Adobe Target Premium vs. Standard,” A4T vs. non-A4T, Activity Types, Activity Sources, etc… Organizations use this data to manage live Activities and for all historical tests. While this is powerful (and very helpful for these two new MiaProva clients), the strategic value lies in tying the organization’s internal data to their optimizations efforts.

With Adobe Target, organizations have the tools they need to analyze individual tests. Organizations can go into Adobe Target and understand how the test impacts any metrics configured in Adobe Target. Lift, Confidence, segments, Success Metrics, etc…..all the great things you need to analyze an individual test.

And with, Analytics for Target (A4T), the analysis capabilities are exponentially more powerful by automating the inclusion of any metric, full use of organizationally accepted segments, metrics, calculated metrics, etc… Organizations can also look at all Activities in aggregate and use Activities and Activity Experiences as Segments which is a massive capability in and of itself that can’t be understated.

What is missing, though, is all the great data that is driving the Activity or test, to begin with. That is where MiaProva comes in by automating the process of mapping data from test tickets or test design documents to the rich data collected and managed by Adobe. MiaProva does this by way of MiaProva Programs – customize templates for their test design documents.

MiaProva Programs allow organizations to ‘dimensionalize’ any internal data related to tests just like MiaProva is ‘dimensionalizing’ the Adobe data like how we are distinguishing Activities that are part of ‘Adobe Target Premium.’ The use cases are limitless, but here are some that are being used by our clients today:

  • What idea source is leading to the greatest number of tests? the greatest lift against metric x, y, or z?
  • How are the tests distributed across organizational locations?
  • What audiences are being used across test locations?
  • How does our feature flaging solution (that we capture via an Adobe Analytics eVar) compare in terms of impact to revenue compared to tests that were executed via Adobe Target?
  • What value is Adobe Target Premium giving us? What is our win rate with Premium Activities vs. Standard Activites?
  • What is our ratio of Premium Activites vs. Standard Activities? What business units are seeing value from Premium and which ones are not using Premium?
  • What Product Managers are runing the most tests?
  • What is the distribution of Activities to Analysts?
  • Are HIPPO driven tests as effective as data-driven tests?
  • What is the ratio of Adobe Target ‘personalization’ efforts (pushing the winner or just targeting content) to actual tests

and we can go on and on. Any data that an organization would like to use is available for management for any Live Activities and all historical ones. This data can come by way of our APIs, JIRA integration, or as users supply tickets as part of their test management process.

Our approach to macro-analysis of optimization programs has enabled organizations to learn fascinating things and exponentially increase the realized value of investments in optimization. It is this approach that these two new organizations to MiaProva started to ask, “Why are we paying for Adobe Target Premium”?

Automated Personalization Playbook

Both of these new MiaProva organizations reached out to us to better understand what they saw in terms of our Program Analysis regarding their investments in Adobe Target Premium. It turns out the value they were realizing was non-existent or low was because they were not using the Activity features that come with Adobe Target Premium (Automated Personalization, Recommendations, and Auto-Target) correctly or in a minimal capacity despite licensing them for a considerable period of time.

Diving deeper into their use cases and their understanding of the Premium Activity types, it became clear that their understanding of Automated Personalization, how it worked, and how best to apply it was incorrect or less than ideal. We dove into the other Premium Activity types, but the investments made for Adobe Target Premium were tied to Automated Personalization.

We tried to point them towards documentation that could help, including a co-authored book on Adobe Target. Still, there was nothing out there that provided a soup-to-nuts approach to how best to use Automated Personalization. So, for these two organizations, and anyone else that may want to learn about Automated Personalization, we are committing ourselves to author what will be referred to as the Automated Personalization Playbook.

Give us a follow below to follow this series over the next month or two automatically:

This Automated Personalization Playbook will go very deep into what Automated Personalization is, how to give you the best chance to ensure success, and outline the best-in-class:

  • Framework
  • Execution
  • Strategy
  • Analysis

Stay tuned, as this will be a pretty powerful tool for those organizations that license Adobe Target Premium. It will also be exciting to possibly report on the change in ROI of these two organizations if they can adapt the strategy as outlined in this playbook!

Automated Personalization for the win

Another organization that leverages MiaProva is seeing significant ROI with investments with Adobe Target Premium. This company had applied a macro-view of optimization way before adopting MiaProva, which was helpful, but now MiaProva automates much of this effort.

This organization invested more than $3MM on an internally built engine that sits on top of its content management system. This system enabled business teams to establish rules for sections of their home page – very similar to how Adobe’s Experience Manager works. The team made heavy use of this solution, leading to significant increases in engagement and impact on key organizational metrics.

In an ongoing effort to continue optimizing, they tested this solution against Automated Personalization (using strategies outlined in the above-mentioned playbook). This effort lasted more than six months and spanned dozens of iterations. Without going into the details, here were the high-level results:

This effort clearly showed that humans are clearly smarter when it comes to applying rules to digital experiences based on data. This effort also showed the value of removing the cognitive bias that humans bring to decision-making and the value that AI could bring.

This ROI is managed by MiaProva and clearly visible in the MiaProva dashboard as a target that they continue to try to hit. This visibility has enabled this organization to demonstrate the value of automation, which has led to its operationalization to several other key areas for its digital consumers.

MiaProva Activity Sources

We added something pretty significant to MiaProva this week. Activity Sources.

As you may know, MiaProva is the only Optimization Management Solution to integrate with Adobe Target and Adobe Analytics fully. Any Activity type or optimization effort in Adobe Target is fully available for management within MiaProva. Any of your Adobe Analytics Segments, Metrics, Calculated Metrics, Events, etc….are also fully available as well. We’ve expanded our deep integration with Adobe I/O to now give our users Activity Sources.

Activity Sources represent different sources for optimization or testing related events. Adobe Target is a wonderful example of an Activity Source, and now MiaProva has broadened Optimization Management to include any data from any dimension in Adobe Analytics.

At MiaProva, our most successful customers have shown us that:

  • Data collected and reported on should have more of an optimization focus with ROI and learning management
  • Considerable opportunities exist within Adobe Analytics to expand on optimization and personalization over and above Adobe Target
  • Organizations need a centralized platform to manage all of their optimization efforts along with the efforts that stem from their testing and optimization solution

Adobe Analytics Dimensions

Adobe Analytics provides organizations with 75 Props, 250 eVars, and 3 List Variables. These variables allow organizations to report and analyze any data that is associated with them. MiaProva now treats each of these as configuration sources for Optimization Related efforts.

These additional sources also inherit the same MiaProva benefits that Activities from Adobe Target get including:

  • Real-time and automated reporting
  • Custom alerting and notifications based on any conversion event in Adobe Analytics based on defined thresholds and confidence levels
  • Centralized management with any other source, including Adobe Target
  • Program Dashboards and ROI models
  • Automatic creation of a knowledge library
  • Collaboration and risk mitigation
  • And everything else available within MiaProva…

Given the limitless ways organizations can use their Adobe Analytics Variables, the opportunities to treat the data collected by them like optimization initiatives are essentially limitless. Here are a few examples to give you a sense of the power this Activity Sources feature enables:

  • All acquisition initiatives (SEM, Email, Affiliate, etc.….) – huge opportunities here!
  • Product Finding Method eVar
  • Form type if B2B
  • Custom feature flagging
  • Internal server-side testing
  • Loyal program management
  • Page Categories
  • Search
  • Custom multi-armed bandits
  • Other testing solutions

Looking to the future, as organizations adopt and make use of Adobe’s Experience Platform (AEP), this Activity Sources capability by MiaProva will be even more valuable given the power AEP has to collect and manage data from many sources. We’ve built this capability for today with the future in mind, given the considerable opportunities that exist with Adobe’s Journey Optimizer.

Configuration in MiaProva

Organizations can easily configure Activity Sources right within MiaProva’s Admin Console as seen below.

MiaProva Activity Sources Configuration

Here organizations can add any of their Adobe Analytics dimensions (eVars, list variables, sProps, etc…) as an Activity Source.

Simply provide a name, the dimension ID, and a value pattern and MiaProva will immediately take over and provide management of these sources along with your Adobe Target Activities!

Below highlights how one can Access or manage these different sources within MiaProva’s Live Dashboard.

Sample Ratio Mismatch (SRM)

Sample Ratio Mismatch (SRM) is a thing, a real thing that optimization programs need to concern themselves about. At a high level, SRM means that the variants (experiences) of an A/B test have a traffic distribution (sample) that differs in a statistically significant way from what you expected the allocation of traffic or the sample to be.

This blog post and this blog post are great as they dive into what it is, how scary it can be, and what to do about it. Please do check out both those posts as they drive home just how important of a topic this is for optimization programs.

Sample Ratio Mismatch and MiaProva

About 5-6 weeks ago, MiaProva quietly released a new Alert to our array of alerts we provide for testing programs. This Alert will notify MiaProva customers that use Adobe Target automatically when SRM has been detected. This approach automatically applies a Chi-Squared calculation to every single A/B Activity currently Live in their Adobe Target accounts.

When MiaProva detects these issues, we will immediately notify stakeholders via email and highlight the Alert in the Live Dashboard and the daily Aggregate email that summarizes all Activity-related Alerts.

MiaProva Alert Management Dashboard

In the last month, we have had SRM detected a handful of times across many hundreds of Live Activities from different organizations. With SRM, the most important thing is identifying the root cause of the mismatch. Two out of the five are still being investigated, but the other three were quickly diagnosed. The statistically significant issues were:

  • During QA, an internal team became members of one of the variants and maintained that membership when the test went live. Incorrect targeting used in the Activity setup further exacerbated this SRM issue.
  • A bot had gotten into one of the variants.
  • Improper use of the Advanced settings upon Conversion, the selected Action below, combined with the audience criteria, caused an imbalance of sample distribution.
Adobe Target Conversion Advanced Settings

Had MiaProva not detected and alerted people to this key issue, organizations could have found themselves in a situation where they were making decisions on statistically invalid test results. Furthermore, because MiaProva detected SRM, the organizations could remedy the underlying causes, thus preventing additional reporting issues.

Below represents what SRM alerts look like in context to Live Activities that are from Adobe Target. MiaProva also provides an SRM calculator with each Live Adobe Target Activity that is an A/B Activity. Automated Personalization, Experience Targeting, and Auto-Allocate Activity types have logic to promote differences in sample distribution across Activity variants. MiaProva provides this service for all A/B Activities, Multivariate Activities, and Adobe Recommendations.

MiaProva alerts on Live Dashboard

And lastly, here is what the calculator looks like and how users can adjust their Expected Visitor distribution as needed.

MiaProva SRM calculator

Offer Impression Capping

Many optimization programs find themselves having a requirement to limit the number of times an offer from Adobe Target gets shown to visitors. The offer could be anything really but this type of ‘offer capping’ is usually applied to overlays or modals of some sort. Organizations would like to have the offer displayed two or three times to the user and if the user doesn’t interact, stop the offer from showing.

Here we dive into a generic profile that sets an impression value independent of any activity or test. Then we show how to use the same logic at an Activity or offer level.

Blog point referenced: https://analyticsdemystified.com/testing-and-optimization/how-adobe-target-can-help-in-the-craziest-of-times/

Chrome Extension: https://www.miaprova.com/blog/adobe-target-chrome-extension-brought-to-you-by-miaprova/

Code used:

adobe.target.trackEvent({'mbox':'activity1234'})

if (mbox.name == 'activity1234') { return (user.get('1234') | 0) + 1;}

MiaProva now provides fully customizable test ticket form – by teams!

At the earlier stage of optimization programs, MiaProva provides a standard test ticket in-take form that is based on industry best practice. Our hope is to guide users to carefully consider all factors that can impact their tests downstream.

As most of our clients reach a level of maturity and sophistication in their testing program, the need to allow flexibility has become clearly pressing. MiaProva is pleased to share that we just released a new capability, where different teams in one organization can setup different test intake forms that is most relevant to their individual needs.

Let’s walk through in some details!

To create a new test ticket form, the test manager with admin access can go to the new TEAMS tab under admin console. Here you can setup team specific work streams, as well as customizable ticket intake form.

You can set default form for the ‘submit ticket’ page to load, as well as create and customize any team-specific new form.

MiaProva’s ticket goes through different states in its life cycle. Here are the key states and in which order they occur:

Intake > Submitted > Under Development > Awaiting Summary > Finalized

You can decide if a specific field is required on each state, or if it’s visible on a given state. Please note however since these states are sequential in nature, MiaProva would require you to enter the data in a prior state IF that field is required in the subsequent state.

In addition to add/remove fields from a default form, you can also add additional MiaProva suggested fields or customize your own fields.

To add a MiaProva pre-configured, most frequently used field:

To add a new custom field, first click the ‘add field’ button, then:

Once you finished configuring an existing form or create a new one, find it as the first item on any ticket submission form. Here is an example under the ‘submit ticket’ section:

On this form, choose the form that you wish to use, and all your custom fields will show up accordingly. Note that this form will govern the remainder of the ticket life cycle! So consider carefully please when you set the ‘required’ and ‘visibility’ options when customizing the form. For instance, if you wish to have a simple form to encourage ease of submission, but want to provide very rich test information in the test summary stage, you might want to consider setting ‘test hypothesis’ as NOT required on the input form state, but add it as ‘required’ on the ‘Awaiting Summary’ and ‘Finalized’ stage.

A note about filters:

Note that with these changes, we added custom filters by team as well – since filters are defined by the fields specified for an individual team. Here is an example:

A note about Jira users’ integration with MiaProva

If your organization uses Jira and has an integration with MiaProva, please note that the form that you set as ‘DEFAULT’ will be the one used to integrate with Jira. This is because Jira does not support the concept of different intake forms by different teams.

Page 2 of 4