Category: Optimization Management

Why are we paying for Adobe Target Premium?

On August 12, 2021

In Adobe Target, Features, Optimization Management

Optimization Management is so much more than managing the intake and output of tests. A true Optimization Management Platform enables the organization to monitor tests or groups of tests and answers the big questions about their testing program as a whole and their investments in resources and technologies.

MiaProva automatically stitches the organizational meta-data related to Activities to data from Adobe Target and Adobe Analytics. This approach allows organizations to use this data to analyze and manage their tests and what offline variables are impacting the program – similar to how organizations use eVars and list variables in Adobe Analytics.

One of the many ways organizations use this is to understand how Adobe Target Premium is adding value to their Optimization Programs. MiaProva automatically classifies Activities that are specific to Adobe Target Premium as a dimension for Program Analysis. Doing so exposes a potential problem that needs addressing if limited or no value is being realized or creates an opportunity to expand on any value that is being realized.

MiaProva just onboarded two organizations that have made significant investments with Adobe and are heavy users of Adobe Target. Both organizations have launched more than 100 Activities launched since January of 2021. While mapping test tickets or test design documents to Adobe Activities to put all that data to life, MiaProva was able to immediately highlight for both of these organizations that they were not receiving the amount of value they should be with Adobe Target Premium.

In this blog post, we are going to highlight how we were able to identify this opportunity (and others) for these two organizations, what is being done to change things, and a great example of another MiaProva client who sees considerable value in Premium and MiaProva has used the ROI models to expand on their use of the Automated Personalization.

Optimization Program Macro-Analysis

MiaProva automatically creates dimensions related to the Adobe data such as “Adobe Target Premium vs. Standard,” A4T vs. non-A4T, Activity Types, Activity Sources, etc… Organizations use this data to manage live Activities and for all historical tests. While this is powerful (and very helpful for these two new MiaProva clients), the strategic value lies in tying the organization’s internal data to their optimizations efforts.

With Adobe Target, organizations have the tools they need to analyze individual tests. Organizations can go into Adobe Target and understand how the test impacts any metrics configured in Adobe Target. Lift, Confidence, segments, Success Metrics, etc…..all the great things you need to analyze an individual test.

And with, Analytics for Target (A4T), the analysis capabilities are exponentially more powerful by automating the inclusion of any metric, full use of organizationally accepted segments, metrics, calculated metrics, etc… Organizations can also look at all Activities in aggregate and use Activities and Activity Experiences as Segments which is a massive capability in and of itself that can’t be understated.

What is missing, though, is all the great data that is driving the Activity or test, to begin with. That is where MiaProva comes in by automating the process of mapping data from test tickets or test design documents to the rich data collected and managed by Adobe. MiaProva does this by way of MiaProva Programs – customize templates for their test design documents.

MiaProva Programs allow organizations to ‘dimensionalize’ any internal data related to tests just like MiaProva is ‘dimensionalizing’ the Adobe data like how we are distinguishing Activities that are part of ‘Adobe Target Premium.’ The use cases are limitless, but here are some that are being used by our clients today:

What idea source is leading to the greatest number of tests? the greatest lift against metric x, y, or z?
How are the tests distributed across organizational locations?
What audiences are being used across test locations?
How does our feature flaging solution (that we capture via an Adobe Analytics eVar) compare in terms of impact to revenue compared to tests that were executed via Adobe Target?
What value is Adobe Target Premium giving us? What is our win rate with Premium Activities vs. Standard Activites?
What is our ratio of Premium Activites vs. Standard Activities? What business units are seeing value from Premium and which ones are not using Premium?
What Product Managers are runing the most tests?
What is the distribution of Activities to Analysts?
Are HIPPO driven tests as effective as data-driven tests?
What is the ratio of Adobe Target ‘personalization’ efforts (pushing the winner or just targeting content) to actual tests

and we can go on and on. Any data that an organization would like to use is available for management for any Live Activities and all historical ones. This data can come by way of our APIs, JIRA integration, or as users supply tickets as part of their test management process.

Our approach to macro-analysis of optimization programs has enabled organizations to learn fascinating things and exponentially increase the realized value of investments in optimization. It is this approach that these two new organizations to MiaProva started to ask, “Why are we paying for Adobe Target Premium”?

Automated Personalization Playbook

Both of these new MiaProva organizations reached out to us to better understand what they saw in terms of our Program Analysis regarding their investments in Adobe Target Premium. It turns out the value they were realizing was non-existent or low was because they were not using the Activity features that come with Adobe Target Premium (Automated Personalization, Recommendations, and Auto-Target) correctly or in a minimal capacity despite licensing them for a considerable period of time.

Diving deeper into their use cases and their understanding of the Premium Activity types, it became clear that their understanding of Automated Personalization, how it worked, and how best to apply it was incorrect or less than ideal. We dove into the other Premium Activity types, but the investments made for Adobe Target Premium were tied to Automated Personalization.

We tried to point them towards documentation that could help, including a co-authored book on Adobe Target. Still, there was nothing out there that provided a soup-to-nuts approach to how best to use Automated Personalization. So, for these two organizations, and anyone else that may want to learn about Automated Personalization, we are committing ourselves to author what will be referred to as the Automated Personalization Playbook.

Give us a follow below to follow this series over the next month or two automatically:

This Automated Personalization Playbook will go very deep into what Automated Personalization is, how to give you the best chance to ensure success, and outline the best-in-class:

Framework
Execution
Strategy
Analysis

Stay tuned, as this will be a pretty powerful tool for those organizations that license Adobe Target Premium. It will also be exciting to possibly report on the change in ROI of these two organizations if they can adapt the strategy as outlined in this playbook!

Automated Personalization for the win

Another organization that leverages MiaProva is seeing significant ROI with investments with Adobe Target Premium. This company had applied a macro-view of optimization way before adopting MiaProva, which was helpful, but now MiaProva automates much of this effort.

This organization invested more than $3MM on an internally built engine that sits on top of its content management system. This system enabled business teams to establish rules for sections of their home page – very similar to how Adobe’s Experience Manager works. The team made heavy use of this solution, leading to significant increases in engagement and impact on key organizational metrics.

In an ongoing effort to continue optimizing, they tested this solution against Automated Personalization (using strategies outlined in the above-mentioned playbook). This effort lasted more than six months and spanned dozens of iterations. Without going into the details, here were the high-level results:

This effort clearly showed that humans are clearly smarter when it comes to applying rules to digital experiences based on data. This effort also showed the value of removing the cognitive bias that humans bring to decision-making and the value that AI could bring.

This ROI is managed by MiaProva and clearly visible in the MiaProva dashboard as a target that they continue to try to hit. This visibility has enabled this organization to demonstrate the value of automation, which has led to its operationalization to several other key areas for its digital consumers.

Adobe Target and Customer Data Platforms (CDPs)

By Brian Hawkins

On April 17, 2021

In Adobe Target, Optimization Management

If your organization leverages Adobe Target for their optimization and personalization efforts and you have not joined your internal Visitor ID to Adobe Target’s visitor ID, consider making that a top priority. This capability has existed for quite some time but is now getting a renewed focus mainly due to ITP, browsers like Chrome are changing how cookies are managed, and Customer Data Platforms (CDPs).

Joining your internal Visitor ID to Adobe Target’s ID takes just a few minutes to complete, costs you absolutely nothing, immediately creates cross-device test coordination, and opens the door wide open to use ANY (except PII, of course:) of the data that you have mapped to that ID. We recommended this blog post or this article by Adobe to see how to do this.

Adobe Target 1st party data, 2nd party data, and 3rd party data

This effort is, by far, the most valuable and highly strategic component of Adobe Target. Doing this converts your optimization program from a 2nd Party one to a 1st Party one where all the data is based on YOUR organization’s ID, not just Adobe’s. All of your Adobe IDs across your various browsers and devices will now map to your organization’s 1st Party ID.

Once this exercise is complete, your organization has a 1st-party optimization program which means ITP and changes to cookies are managed within browsers will not impact test execution and analysis. In fact, a 1st-party optimization program not only future proofs investments in testing and personalization but also enables your organizations to expand their efforts considerably with customer journey optimization.

Okay, so now you have the 1st Party ID in place, now what…

Configuring Adobe Target to become a 1st-Party Optimization program is the easy part. Historically, and still quite common, the more challenging part is figuring out where internally at your organization the 1st Party data is located that is mapped to that visitor ID that you now have mapped to Adobe Target. Then once you know that, the challenge then becomes how best to get it to Adobe Target.

The where…

Organizations vary considerably in terms of where this rich 1st-Party data exists. The data could be located across several different systems and spread across an array of different teams. The data might be in data-lakes, data layers, cookies, completely offline, and in some cases not joined to the Visitor ID you used but rather other definitions of a Visitor ID. Users of Adobe Target, those that execute the Activities, may also not have access to the systems or teams that manage this 1st Party data.

Data compliance is another important factor concerning where the data is located. Data compliance is the practice of making sure that the data being managed is guarded against misuse. Depending on where the data is located, it may or may not be data compliant – safe to use for testing and personalization within Adobe Target.

As you can see, finding safe-to-use data can be a challenge. Thankfully, some organizations have overcome this challenge and established governance and processes to get the data. This then leaves them with the “how” to get this data into Adobe Target for their personalization and testing needs.

The how…

Once you know where the data is, Adobe Target provides quite a few options to get the data into Adobe Target. Depending on the format your data is in, you may also have to modify the format to leverage one of the options available. Here are the main mechanisms organizations get data into Adobe Target for personalization and targeting Activities:

Adobe Target API – here you can update single Profile attributes or use Adobe Target batch files to upload as many as 1,000,000 Visitors per day (I know of several organizations that exceed this considerably and it works wonderfuly)
Adobe Target adbox – this is an old-school tool that many of us old-timers use from time to time. It is simply a pixel that passes data to Target. Very helpful from taking data from email platforms and passing it to Adobe Target upon email open.
Client-side or Server-side via parameter value pairs – just like your Analytics tags collect data that Adobe Target mbox calls as well. This is a very popular approach given the use of local stroage and datalayers and Adobe Target’s ability to easily consume that information.
Adobe Analytics Customer Attributes – those organizations that make strategic use of Customer Attributes are using this approach to very creatively to get 1st party data into Adobe Target. Adobe converts these attribute to Adobe Target Profile Attributes automatically.
Adobe Audience Manager – within Audience Manager, organizations can create segments as they wish and share them to the Experience Cloud, which makes them available within Adobe Target. Quite a few of the organizations that use MiaProva, leverage this approach.

As you can see, the highways are built and are ready to use. Adobe has done a fantastic job giving organizations different options to get the data that they need to Adobe Target for personalization and testing. Despite these mechanisms, organizations still struggle to establish the infrastructure to automate and maintain the data transfer despite the considerable value.

The value…

Out of the clients that use MiaProva and the organizations that we work with at Demystified, it is safe to say that only about 45% of them leverage the functionality that enables 1st Party optimization programs. The reality of it all is that while the functionality is there and the value is massive, organizations often struggle to take advantage of these capabilities. Learning the capabilities, connecting the pipes, establishing process and governance, etc… are barriers that many organizations can’t overcome.

Those organizations that have been able to overcome the barriers realize significant value. Here are just a handle of many examples of organizations and what they are doing with their 1st Party Optimization Programs:

For a $1.2B retailer, weekly (on Sundays), they use Adobe Target’s batch processing for Profile attributes processing in-store transaction data so they can be used in key Activities by the various testing teams for customer journey optimization.
A $900M retailer, before each email blast, they pass key profile attributes via a pixel call to Adobe Target that is specific to that email in the event the visitor “view throughs” to the website vs. clicking through. This effort was instrumental to coordinate pricing threshold messaging offline and online.
For a $7B retailer, 1st Party data is made available as Response Tokens so that their CMS can use the data for content decionsing and rules.
For finance, a $20B financial institution, using Audience Manager and Adobe Target batch APIs is allowing them to give Automated Personalization very rich 1st Party data for modeling and content decisioning. Additionally, considerable personalization efforts via Target are rooted in this data.
A $39B financial instituation is using client-side and server-side parameter passing Tag Management for very advanced use of Automated Personalization along with rich analyis via A4T. Additionally, this approach enables their internal call centers to mimic experiences of the digital consumers by way of the cross-device test coordination.

The challenges around “where” the data is and “how” to use it are not specific to personalization and optimization efforts. The increasing demand to make data actionable, connect multiple data sources, and provide data compliance expands well beyond testing to all marketing and data science components.

This perfect storm has created a vacuum that is being filled by something called Customer Data Platforms (CDPs).

CDP

What is a CDP? This is not a simple answer by any means, as CDPs providers have a wide range of features and capabilities that align and manage consumer data for organizations. CDPs can make the “where” and the “how” considerably easier than it has to be, and some CDPs can automate those efforts.

We are not going to go into all that a CDP is here. At a high level, a CDP is all about managing an individual customer with a single profile. CDPs have visitor stitching algorithms, provide data compliances, and make the data available to different departments and solutions like Adobe Target, for example.

ACTIONIQ

MiaProva customers vary considerably in terms of CDP use with testing and optimization. Those that do can automate and accelerate their use of 1st Party data for testing and optimization. Our clients work with several different CDPs, but I wanted to call out ACTIONIQ specifically because they work with Adobe Target to bring the 1st party data to life.

I’ve highlighted above how organizations have to establish processes and governance around the “where” and the “how”, but here is how ACTIONIQ automates that process seamlessly, specifically for Adobe Target, so that testing teams can focus on strategy and execution.

Real-time syncing – as this first-party data gets updates, these guys update the Adobe Target Profile as events take place across other channels
Batch synching – weekly, every 10-days, Sunday nights, they can automate the process of porting Adobe Target Profile attributes for personalization and testing. Much easier than gathering a file internally and uploading it from your PC.
Porting Adobe Target Activity Meta-data to other solutions. CDPs shouldn’t be limited to feeding Adobe Target data about the digital consumers, but they can also consume what Activities or tests your digital consumers are members of. Consider how helpful it would be for internal Call Centers to have visibility into this information? Or to coordinate these onsite personalizations with initiatives with Social or with Acquisition initiatives?
It is related to #3 above, impression management. If you are using Adobe Target to show digital consumers an offer through an Activity, you can control in Target how many times they see it. This is better served in a CDP where you can coordinate impressions of ‘offers’ or content at a macro level across channels and mediums.

MiaProva and CDPs

The customers of MiaProva that use CDPs as part of their personalization and testing efforts are increasingly using the data the facilitate Customer Journey Optimization. Customer Journey Optimization is where organizations have testing roadmaps with targeted testing and personalization, advancing customers across their various customer journeys.

MiaProva supports those efforts by way of our MiaProva workstreams, where we automate the management of metrics, segments, tickets, activities, and ROI based on these workstreams – making it more efficient to advance the throughput of these efforts.

Customer Journey Optimization

Customer Journey Optimization

By Brian Hawkins

On February 27, 2021

In Optimization Management

All of the more advanced and sophisticated optimization programs did not start that way. How they started and their path to awesomeness most certainly varied considerably. Despite these varying paths, there were key components of their foundations that were constants: scalable standardizations with risk mitigation, clear goals with alignment to organizational KPIs, and program visibility with accountability.

Those foundations have given these optimization programs the nimbleness to adjust to changes in organizational priorities, technology advancements (and deprecations), and the changing landscape of available opportunities. Without this agility in optimization program management, the path to awesomeness would not be possible.

At MiaProva, we get to see it all, the really good, the not-so-organized or focused, the up and coming, and the rockstars. No matter where their program is, MiaProva brings the foundation for any organization to find its own path to awesomeness.

Some of our advanced customers who have found their awesomeness are using MiaProva to advance their programs through Customer Journey Optimization further. Given the value they are realizing, we thought we would share what that looks like and the features in MiaProva that they are using.

Customer Journey Optimization is when organizations have optimization or testing roadmaps targeted to the different stages that their digital consumers are in with the goal to advance them to the next stage of the customer journey. For most organizations, A/B testing is often targeted to a specific location like a landing page or home page. With Customer Journey Optimization, A/B tests are running simultaneously to independent audiences tied to where they are in the journey.

The Path to Customer Journey Optimization

Historically, organizations often judged their program’s sophistication by the test approach (A/B, Multivariate, Multi-Armed bandit, etc…) and the targeting they were applying to their tests. The large bubbles in the image below represent this mindset broken down by Value, Goals, Data Leveraged, and Opportunity.

Customer Journey Optimization is all about the Data Leveraged, centering your optimization program around your organization’s 1st Party ID – your customer’s obfuscated ID.

When organizations make this strategic shift and steer their testing efforts around getting an ID (Unknown to Known), they can start to build that relationship and advance them down Customer Journeys deepening that relationship.

In the image below, test types are just the tool to target personalizations and learn, through test results, the ideal personalization for those consumers at that stage.

MiaProva and Customer Journey Optimization

MiaProva provides a variety of tools to allow organizations to adopt strategies like Customer Journey Optimization easily. The one feature that has been the most helpful according to our customers is our test management via our kanban board.

The image below is our kanban board in action. We have everything from idea prioritizations, test submissions, tests in development, live tests, and those awaiting analysis—a full view across the various stages.

In Red Box 1, we have the MiaProva Journeys that have been customized for Customer Journey Optimization. In Red box 2, we have tasks specific to those tickets, and in Red box 3, we have live data with metrics associated with that journey and test.

This video provides a nice overview of how the kanban works and how organizations are using it for program scale, accountability, and visibility.

Analytics for Target (A4T) and Optimization Programs

By Brian Hawkins

On November 9, 2020

In Adobe Analytics, Adobe Target, Optimization Management

The other week, I had the pleasure of joining Adobe on a webinar to dive into Analytics for Target (A4T) and how it can be a powerful tool for optimization programs.

MiaProva makes very heavy and strategic use of A4T as part of our Optimization Management Application and within our Chrome Extension. Our customers that leverage A4T are realizing increased test throughput, visibility into metric and test interaction, and are gaining significant returns on their investments in optimization. We are also seeing our A4T customers adopt personalization and customer journey optimization at a much higher rate than our non-A4T customers.

In this webinar, we shared how some of the MiaProva magic can be leveraged right within Analytics Workspace because of A4T!

Here is what to expect as part of the agenda:

A small case study on how organizations can leverage Segments from Adobe Analytics to empower Automation within Adobe Target. Specifically, we looked at Adobe Target Insights Reports across 4 different organizations. We looked at the traffic volume, lift, and what data the models were focusing on. We then highlighted how some organizations that took the time to create and share segments from Analytics they deemed to be of value, resulted in the AI models in Adobe Target to automate at a quicker rate and identity what visitor attributes were key to making that happen.

MiaProva magic and the efficiencies of scale. Here was showed how A4T introduces mechanisms to allow the organization to align around metrics and audiences in a manner that is not possible in a standalone optimization or personalization solution.

Macro-analysis of Optimization programs. A4T enables organizations to not only examine specific or a group of activities from Adobe Target but also enables aggregate analysis of the program as a whole and whether you are making full use of Adobe Target

And lastly, we close things with a case study of how an organization made strategic use of A4T to analyze a highly visible series of Category Affinity Activities taking place. This organization learned many interesting things but the most valuable component of the learnings was taking a key audience that partook in those Activities and shared them back to Adobe Target via a segment for a more personalized experience.

We also shared a series of action items or tips that you can use to familiarize yourself with A4T and how it can be of value to your optimization and personalization efforts.

Adobe Target technical interview questions

By Brian Hawkins

On August 5, 2020

In Adobe Target, Optimization Management

MiaProva facilitates scale. In fact, we have found that, on average, our customers are running ~30% more tests than before they were using MiaProva. Because of this scale, organizations often find themselves in need of additional resources to support their optimization program.

This was the case for one of our valued customers. They reached out to us to see if we could provide some technical questions to gauge the applicant’s technical capabilities in regards to Adobe Target. This role at this organization requires much more of a technical background than a typical end-user of Adobe Target due to the environment and the advanced nature of the integrated components in place.

So we did just that and put together the below list of questions and thought that others might find them interesting or helpful.

Technically focused questions

1. Can you walk me through what this code is doing?

<style id="flickersuppression" type="text/css">
    #LegacyVersion {visibility:hidden !important}
</style>
    
<style id="stop" type="text/css">
    #slm-master-content-col-rightrail {visibility:hidden !important}
</style>
    
<script type="text/javascript">
    
    (function($){var c=function(s,f){if($(s)[0]){try{f.apply($(s)[0])}catch(e){setTimeout(function(){c(s,f)},1)}}else{setTimeout(function(){c(s,f)},1)}};if($.isReady){setTimeout("c=function(){}",100)}$.fn.elementOnLoad=function(f){c(this.selector,f)}})(jQuery);
    
    var pname = utag_data["page_name"];
    
        if (pname=="HomePage") {
    
            $('#RedesignVersion').elementOnLoad(function(){
    
                $('#LegacyVersion').hide();
                $('#RedesignVersion').show();
            })
        }
    
    $('#flickersuppression').remove();
    
</script>

2. Please create a redirect offer manually vs. using the built-in redirect offers in Target.

3. How can you set a user. profile attribute based on a mbox parameter?

4. How can you create mutually exclusive groups using profiles?

5. Please explain what response tokens are and what benefit they would bring.

6. High-level, what is A4T and how does A4T work?

7. What is the difference between Target’s Auto-Allocate and Experience Targeting Activities?

8. What is the difference between an Analytics List Variable vs. a regular eVar?

9. Can you provide an example of an offer that executes on page load vs when an element has loaded into the DOM?

10. How can you disable Adobe Target?

Optimization in our remote world

By Brian Hawkins

On July 2, 2020

In Optimization Management

Testing is not easy by any means. In fact, it is arguably quite challenging. Running and managing an optimization program is even harder. If you think about all that happens behind the scenes, it is pretty overwhelming.

MiaProva was established to make testing and program management easier. MiaProva brings automation, standardization, and a variety of tools that allow organizations to scale effectively and quickly. In fact, we have found that our customers, on average, have experienced a 30% increase in test throughput since adopting MiaProva.

Over the last few months, we have gotten some positive feedback about how MiaProva has been especially helpful during this new world that we all live in. All of our customers immediately found themselves working remotely (many for the first time) and they were just as busy if not more.

And now, the everyday challenges around testing and running optimization programs are additionally taxed with this new remote world. Especially tough given testing teams touch so many different parts of an organization.

MiaProva and how we help…

We thought we would highlight some of the utilities in MiaProva that have been especially helpful for organizations as of late and broadened the adoption of MiaProva beyond testing teams to the broader organization.

#1 – Communication

MiaProva keeps those that want to be in the know, in the know. As events related to the test happen, MiaProva will automatically send alerts via email and slack to key stakeholders.

One of our clients shared that the automatic communication from MiaProva allowed them to cancel a weekly meeting that was used just to keep everyone up to speed on tests. MiaProva has become our conference room with centralized management of everything optimization, she added.

#2 – Security – Test Risk Mitigation

MiaProva provides a security service for all of your tests against any of your metrics in Adobe Target and Adobe Analytics. We also monitor and alert you to other key events such as issues related to the code you are using for your tests.

For those that leverage A4T, simply select in MiaProva, your key segments, metrics, and calculated metrics and we will monitor them all and provide daily updates over and above realtime alerts.

We have a number of great stories about this service saving the day for organizations. We had one client tell us that our monitoring prevented significant financial losses to their organization that would have gone unnoticed for days. Another client told us that a single alert that we sent late on a Friday saved them more than 12 years of MiaProva licensing costs.

#3 – Transparency – test interaction

Our chrome extension is wildly popular (used on average 15,000 times a day) and is completely free for anyone that uses Adobe Target. One of the reasons it is so popular is because of the visibility it provides not only into test interaction but also the Adobe Target Profile.

With many organizations democratizing testing, companies soon find themselves in situations where too many tests are running and overlapping with each other. We recognize that problem and built tools like our swim lane view and chrome extension that help expose these issues for organizations.

This report alerted one of our clients to a major issue. They were unaware of dozens of live tests getting significant traffic that they were unaware of. We have had several clients discover that audiences were overlapping as well and causing issues around test attribution.

#4 – Program Visibility

One of, if not, the biggest challenge organizations have with testing programs is visibility into where things are and the impact the program is making against their institutional metrics. Visibility was one of the primary reasons we built this platform and we solve for it nicely.

On your MiaProva dashboard, you are provided with a quick view of how many tests the program is doing month over month, your program win rate, and a view into where your tests are in terms of status.

We also highlight here the “Featured Test Story” (selectable via the Test Manager) as well as “Latest Happenings”, a real-time and historical display of events related to your tests or your MiaProva tickets.

MiaProva’s “Latest Happenings” are instrumental in helping remote teams stay up to speed on all the moving parts involved with tests. For larger programs or decentralized testing teams, this feature has aided in collaboration and in communication as end users can add comments to tests and tickets which also become part of the feed of events.

#5 – Centralized Knowledge library

The MiaProva knowledge library was the driving factor in starting MiaProva. To this day, many organizations struggle with this. MiaProva automatically curates the knowledge library by simply managing your tests through our process, which in and of itself, brings considerable efficiencies.

A centralized Knowledge Library allows remote teams, large organizations, and complex testing programs a quick and easy way to leverage prior learnings, avoid testing the same concept multiple times, and a mechanism for the organization to learn from the tests as well.

#6 – Live Dashboard

Our very popular Live Dashboard consolidates and standardizes reporting from multiple data sources (Adobe Target, A4T, Report Suites, Virtual Report Suites, internal testing tools, or other 3rd party testing solutions) into a single place. We highlight which tests have met preset sample sizes and enable filtering by Adobe data as well as any data or field used in your MiaProva ticket.

Our approach to the Live Dashboard is built around your test meta-data. We don’t limit your ability to find tests by test name or test type. MiaProva Live Dashboard is completely customizable. For example, if one of your MiaProva ticket fields was “Product Manager”, that field then becomes a filterable or default view which makes accessing tests incredibly easier if your large program has dozens or hundreds of live tests.

We also provide commenting capabilities in the context of test results – very helpful for many reasons but especially helpful for remote testing programs. This feature allows everyone in the broader organization to be part of the conversation, the ability to ask questions or get clarification on tests results. Comments added automatically trigger alerts and send emails to the test managers associated with the test.

Many of our clients have expressed their appreciation of the MiaProva comments feature because it helps avoids countless emails, unnecessary meetings, and aligns key stake holders.

#7 – Crowdsourcing

Testing programs require significant input from the broader organization. This usually involves many meetings, emails, and conference calls to advance things along. Successful testing programs, incorporate crowdsourcing tools to bring efficiencies and to scale effectively.

We the increase of remote teams, crowdsourcing tools are now crucial for testing programs to be successful. MiaProva provides several tools to aid testing programs but the most helpful has been a mechanism to solicit test ideas as seen here:

MiaProva is also heavily leveraged for crowdsourcing votes for test ideas and providing visibility and transparency around the testing process.

#8 – Standardization – A4T, Automated Personalization, Server-side, mobile iOS

If an organization is to be successful in instilling a culture of optimization, the testing program needs to standardize key processes related to testing.

Most organizations struggle with a common approach to test execution. Approaches often vary depending on team, test type, data, reporting, etc….

MiaProva solves this by standardizing and streamlining the approach to test management. No matter where you are testing, what reporting source you are using, or type of test – our model establishes an assembly line approach to test management.

#9 – Reporting

MiaProva standardizes, simplifies, and consolidates test data from multiple sources.

MiaProva provides reporting for all of your live tests using any of your metrics in Adobe Target or Adobe Analytics. Simply specify your key metrics in MiaProva once and by way of our integration with Adobe and we will automatically report and monitor those metrics as they relate to all tests going forward.

Additionally, we realize that not all test data is available in Adobe or by way of API. Our test management process allows organizations the ability to consolidate this offline data to their test results.

For example, one of our financial customers has offline events related to loans that happens weeks after the test was turned off. MiaProva consolidates this data to the test results when it becomes available.

The outcome of our reporting management is our Program Overview. Here we aggregate all of the tests and their impact to key organization metrics – both online and off.

MiaProva’s Program Overview is something is automatically curated and saves organizations countless hours aggregating and aligning metrics.

Our approach has allowed many of our customers to convey to Executives exactly the net impact they bring and to use that data to increase headcount, expand testing, and to advance their testing and personalization strategies.

Many of our clients leverage our Program Overview as part of their Quarterly/Annual Business Reviews (QBR).

#10 – Integrations with Adobe, Slack, and JIRA

MiaProva believes testing, testing solutions, and optimization management tools like MiaProva shouldn’t live in a vacuum. Because of this, we’ve built our platform to integrate with other key solutions like JIRA and Slack given their importance to optimization programs.

MiaProva has a bidirectional integration with JIRA to import and export test tickets. Our popular integration with Slack is heavily used for alerting, live test results, and seeing where tests are in their development lifecycle.

MiaProva didn’t sacrifice security to promote and support an open ecosystem. We are heavily used by some of the most secure organizations out there including many financial institutions around the world and that wouldn’t be possible unless we adhere to very strict security protocols.

Last Time for Lost Time

By MiaProva

On January 10, 2020

In Optimization Management

What is your most valuable, irreplaceable resource in the never-ending quest to engage your customers?

Time.

It passes, it’s gone, and you can never recover it.

An inferior customer experience, a mediocre campaign, lackluster messaging: these cause days and weeks to slip by without gain. New customers go to competitors, and now you have the chore of winning them back. Customer loyalty languishes, along with customer wallet share and lifetime value, and it becomes harder to fund marketing.

Haphazard experimentation eats up irreplaceable time.

This is why you need great test management. Great test management effectively uses every minute for:

coaxing customers away from competitors
finding new customers
more deeply engaging your customers’ trust and attention
gaining deeper loyalty and wallet share
proving your efficacy as a marketing organization, worthy of more resources and greater investment

Of course, none of us is happy about time slipping past, or not getting the most out of every opportunity. How, exactly, are we supposed to seize the minutes, wringing everything we can from them?

The really successful organizations I’ve worked with connect their testing operations with their business strategy. They accomplish this with a proven organizational glue: management. Test Management cements your business strategy goals into your test operations.

MiaProva’s Program Overview indicates at a glance how the program has contributed to business goals. Of course, you have more than one goal, which is why this overview allows you to select which goal to view. You can also select by customer segment, dates, locations, etc. The panel at the bottom of the page displays each test and its impact.

Without test management, your test activity can’t accomplish all that you need from it. Here’s how test management can help you get what you need.

Your Optimization Needs	Do it This Way	Not this Way	MiaProva Test Management Features
Run experiments that have an impact, and really move us toward our goals.	Estimate the potential impact of a test Review past learnings for insights and to avoid re-learning Rapidly retract tests that have a negative impact	Unwittingly re-run past experiments Spend time on tests that will have little impact Let negative tests continue longer than absolutely necessary Spend time on tests that are poorly designed	Idea Quality Score calculated for every idea, based on perceived difficulty and value. Guide to optimization – sample size, meaning, and significance of results; built-in testing best practices Get instant alerts when test experiences break or hurt a KPI Automated knowledge library allows search of past tests, by location, outcome, and impact

Develop stakeholder skills to contribute ideas, analysis, and knowledge to testing, and absorb the learnings	Involve stakeholders in the program’s activities and success. Every experiment tells a story, and it needs to be amplified and shared. Make legends that resonate. Bust myths that obstruct progress.	One or two people think of tests to run, working in isolation.	Crowdsource testing ideas and standardize prioritization. Everyone collaborates to create the best possible testing program. Build and share case studies detailing inspiration, hypothesis, variables, and results. One-click to create and distribute test summary, including design and business impact.

Demonstrate that the testing program is delivering significant benefits.	Have impact metrics at your fingertips.	Rely on notes or screenshots indicating a test “increased conversion by x%,” but can’t quantify the impact on business goals	One source for reporting and modeling that track impact on business goals over a period. Show and Tell how the testing program is contributing to the bottom line, including contribution to each KPI in any period View tests by impact, location, date, KPI goal

Never be caught off guard by test activity Know-how testing is fitting in with customer and marketing activity	See at a glance what’s going on right now, including impacts and results	Look at your notes, and ask colleagues what they have in the process; look at analytics to see if anything unusual is happening	Single Test Management home page shows all tests currently live, under development, or awaiting release Test Management Workflow tracks requests, tasks, prioritization, QA, and status Easy calendar view of all tests and key events

Effective managers can lead organizations and people of any maturity, and MiaProva is just such a manager. Features of MiaProva’s test management help organizations wherever they are on the optimization learning curve. It is your number one strategic asset for continuously improving your customer experience, whatever your organization’s skills and resources.

The most effective programs have broad stakeholder participation, especially for finding and prioritizing test ideas. MiaProva makes it easy to chip in with ideas and opinions. People submit ideas, “like” ideas already submitted, and vote on the ideas they feel will have an impact. MiaProva assigns a Quality Score to ideas based on estimates of effort and return.

Optimization PhDs use MiaProva to:

Communicate their program’s business value and strategic impact
Share stories about customers and experiences with all stakeholders, to increase skills and participation
Review past learnings for insights, for great test ideas, to avoid lost time from re-learning, to improve future tests
Know at a glance the current status of tests in-process and under development, and how those tests impact KPIs

Testing Undergrads use MiaProva to:

Coach them in managing test activity, via alerts on test progress and findings
Create and choose great test ideas
Run tests effectively, with best practices in sample size, timing, and audience
Understand the meaning and significance of test results
Communicate impacts and insights of individual tests and the test program
Come by and sign up for a trial, or to learn a more. You’ll quickly see how to make time work for you.

MiaProva 2018

By Brian Hawkins

On December 31, 2018

In Adobe Target, Features, Optimization Management

Like you, we love data…

As we are closing out the year here at MiaProva, we took a step back to examine some numbers and trends over the last year. Like you, we love data and thought we should share just a few of the interesting tidbits here.

MiaProva Application

22,988 Adobe Target Activities are being managed within MiaProva. This is significant because that means that 22,988 tests or Adobe Target Activities can be automatically included into their Program Overview dashboard for real time visibility into key organizational metrics. These Activities are also actively monitored and are ready for consumption into an organizations easily consumable Knowledge base.

22,988 tests is a HUGE number and we are quite proud of it. This wouldn’t be possible without the automation we’ve created by taping into Adobe I/O. Here is a breakdown of the Activities by type:

A/B Activities: 15,916

XT or Experience Targeted Activities: 6,920

ABT or Automated personalization, Auto-Allocate, Recommendations, MVT, etc…: 152

MiaProva works seamlessly with any test or Activity type used in Adobe Target. It is going to be interesting to see if the ratio of advanced test types like Automated Personalization or Auto-Allocate changes in the months and year ahead. We expect it to considerably and our Chrome Extension should help facilitate that a bit.

MiaProva Chrome Extension

We launched our free Chrome Extension (CE) for all Adobe Target users back in September and the adoption and use has been great. It has been used 252,090 times since the launch up until today, 12/31/2018.

Activities

One of the key components of our CE is visibility into Adobe Target Activities and the interaction of those Activities with one another. In the image below, we can see that on this page (our home page), we have 4 active Activities that we are in at this location and as a Visitor, we are associated with another Activity in a different location.

Having 4 tests running on the same page is definitely not ideal for many reasons including attribution reporting and code conflict. Many organizations run into this challenge especially with more democratized programs and that is why we included this as the primary tab of our CE.

Below is a breakdown of the number of Adobe Target Activities by page over the last three months. As you can see, while the majority of tests per page is one, there are a significant number of test locations with many more than one running. Two test locations were found to have a whopping 15 Activities running!!

Profiles

Adobe Target profiles are arguably the most powerful component to the Adobe Target platform. We wanted organizations to have visibility into what profile attributes were and what values were associated with their Adobe Target Visitor ID (mbox PCID).

The image below represents the profile attributes associated to my Adobe Target ID. Adobe Target users can use this on their site to familiarize what is associated with them, they can edit values, and then can even add new values. Pretty powerful for many reasons but especially helpful for debugging.

MiaProva’s Chrome Extension Profile Tab

While it is not a true representation of how sophisticated a testing program is, the number of profile values does highlight an organizations familiarity with this component of Adobe Target.

Over the last three months, we have found the number of profile attributes varies considerably. The average number of profile attributes was 62.43. The most common number was 14 and the highest number of profile attributes seen was 323!

2019

There is a lot to look forward to in 2019. Optimization and Test Management solutions have proven to be an essential part of any organizations testing program. We are quite proud of our ability to help our clients this year: realize the impact they are making into organizational metrics, incorporate efficiencies and visibility around the entire testing process, and see that not a single test goes to waste with our automatic knowledge base construction.

We are full steam ahead on this mission and wish you all a happy and optimized 2019!

Welcome to the MiaProva Blog!

By Brian Hawkins

On February 7, 2018

In Features, Optimization Management

It has been almost a year since MiaProva came out of beta and was publicly made available to the world. It has been an incredible first year with the adoption of many great customers and several new features added MiaProva based off of customer ideas and trends in the industry!

Introducing the MiaProva blog

With all the exciting things that are going on with MiaProva and testing in general, we decided to formally publish a blog to share these things for our digital consumers and to foster conversations on optimization and test management.

Here is a very fluid list of things you can expect from MiaProva in future blogs

– Features and product updates to MiaProva
– Real Q&A from our customers
– MiaProva Product tips
– Industry related topics
– Fun and geeky things around testing and optimization
– Adobe Target I/O
– Adobe Target Profiles
– Public events that MiaProva will be part of
– and, anything else that we or you might find fun or interesting…..

Thanks for reading and we look forward to continuing the conversation.