Every day at Microsoft, about 50 new controlled experiment treatments (e.g., A/B tests) start, across multiple products, using Microsoft’s ExP platform. These experiments have allowed product teams at Microsoft to evaluate ideas objectively, and make data-driven DECISIONS, accelerating innovation. After sharing some surprising real examples, some worth tens of millions to over $100M annually, we focus on valuable lessons learned over the years. Learn how, and why Experimentation is vital for your organization.

Those who experiment most, innovate best. But running large scale A/B tests is not just a matter of infrastructure and best practices. It requires instilling experimentation at the heart of the decision making process. With that comes the cultural challenges of issues like embracing failure, democratizing results, and more in-depth collaboration. These challenges must be addressed not just with individual teams or contributors, but with leadership as well. In this fireside, we’ll chat with Ya about how experimentation changed the culture at LinkedIn, why it was a necessary transformation, and how to apply what they’ve learned to transforming the culture of decision making in your organization.

One thing that separates excellent product teams from others is how well they use analytics to make decisions. Product teams live in a data-driven world, and the bar is rising. Analytics increasingly offer greater transparency in understanding how your customers are using your product, what’s working, and what isn’t. Come to learn best practices in how to use metrics to optimize your product.

Product management expert Dan Olsen will share advice from his book The Lean Product Playbook on a repeatable process for how to improve your product with metrics, illustrated with real-world examples and case studies.

At LinkedIn, the real measure of operational excellence is defined by two key metrics - Mean time to Detect (MTTD) an issue, and Mean time to Recover (MTTR), which is time taken to triage and fix the issue. The reason why these two metrics are important is that the longer it takes to detect and recover from an issue, the higher the business impact. Getting an alert isn’t enough, because even if a team is made aware of an issue, the company will continue to lose revenue until the team can get to the cause of the alert and provide a fix. ThirdEye, the internal alert & monitoring tool at LinkedIn, was built to perform anomaly detection and root-cause analysis at scale, in addition to determining the business cost of individual alerts.

This session will cover how ThirdEye:

• Employs a combination of data-driven algorithms along with business rules to improve MTTD without compromising on the quality of alerts.
• Performs anomaly detection, metric de-aggregation, and analysis of related events in real-time as the user interacts with it.
• Cross-references dimensional information of events and metrics to present likely explanations for the issue under investigation.
• Helped identify swings in metrics at LinkedIn due to traffic spikes, conflicting deployments, ongoing experiments, and even simple holidays.

In this talk, Major MJ Hegar reveals how to be a catalyst for change on a large scale, be it in your career, at home, or on a national or global level. “When someone tells you that something cannot be done,” she says, “the only way to prove them wrong is to do what they believe to be impossible.”

Her heroic actions in Afghanistan—when she was shot down on a Medevac mission and was pivotal in rescuing her crew and patients, and became the second woman in history to earn a Distinguished Flying Cross with Valor Device—changed the perception of women in the military. And, her historic lawsuit against the Secretary of Defense opened the door for women to participate in ground combat, a significant step forward for equal rights for women in the military. How did she do it? And what can we learn from her experiences? A magnetic speaker, she combines harrowing war stories and powerful lessons on social change with practical advice on leadership, diversity, teamwork, and motivation.

The software industry is currently having a moment. Techniques and tools that have been proven out with technology-forward companies are now being adopted widely across more traditional industries. This adoption is driven, in large part, by these technology-driven companies targeting every industry.

There is a race between the Teslas of the world learning to become a car company compared to the Fords of the world learning to become a software company. This race is driving an enormous wave of innovation as we make techniques that were often reserved for big tech companies available much more widely. In this fireside chat, we’ll dig into Joe’s brain as he unpacks what these techniques are, and how you can apply them in this world where every business in every industry must digitally transform if they want to survive.

Many companies building in house data science teams are doing so because everyone else is doing it. However, not everyone understands how to measure the return on the investment they are making. What should you expect from your data team? In this session, I will focus on the kinds of impact you can expect to have from investing in building a data team. Specifically the talk will focus on how to use data to understand the key activation use case (Atomic Use Case) for onboarding of new users. Additionally, once you have understood the Atomic Use Case, how you can use that to quantify the Lifetime Value of your users.

There are plenty of posts on Hacker News extolling the virtues of CD and how to implement it in your favorite tech stack. But what if you’re not starting from scratch? What if you’ve already got a large code base that the company depends on? What if the way your team worked for the better part of a decade requires a paradigm shift? We all know how open engineers are to change… The best way to grow is to learn from your mistakes, but it’s even better if you can learn from someone else’s blunders. So come with me on a cathartic journey through my team’s CD misadventures. From deployment rollback horror stories to Black Friday engineering nightmares, I’ll detail the good, the bad, and the ugly on our continuous delivery quest.

“Outcomes, not outputs!” is a common rallying call these days. To passionate problem solvers, this message likely resonates. Who wants to work in a Tayloristic feature factory as a glorified ticket taker? Who wants to ship prescriptive deliverables with disregard for the impact of our work (the “outcomes”)? No one!

Unfortunately, the “outcome vs. outputs” narrative and dichotomy is far too simplistic. At its worst, it marginalizes teams, encourages handoffs, and reinforces silos. The reality is rarely that binary. Teams eagerly attempt to realize the promise of outcome-driven approaches but struggle with the details. Tools like OKRs only seem to exacerbate the problem, becoming more of a nuisance and hoop-to-jump-through than a learning/alignment enabler. Even our work visualizations — kanban boards, for example — can obscure how our interventions and “bets” are genuinely related.

In this talk, we’ll explore a more nuanced view of outcomes and outputs through the lens of value creation networks, bets, and promises-in-progress. I will cover:

• Impulse and response — why we tend to overlook the passing of time.
• How to connect our day to day work to the “big picture.”
• A framework to discuss “prescriptiveness.” Finding the sweet spot for your teams
• Why outputs aren’t inherently evil — but separating them from outcomes is.
• Assumption mapping, nested bets, and dueling with uncertainty
• Promises-in-progress: how visualizing the work is not enough.

My goal for this talk is to expand beyond overly simplistic views while staying actionable and grounded in the reality of our day to day work. I’m excited to attend Decisions to learn, collaborate, and contribute!

As engineering teams work to deliver faster, product teams must also work equally as quick to measure the value of what is being delivered. So who owns the responsibility of proving value, and how do they do it? This evolution of the data-driven product manager means a new way of working, and a new set of solutions to enable that work.

Earlier this year, the Product Cloud Alliance was announced, bringing together the most critical of those solutions with the goal of helping product leaders move from being tactical to strategic innovators within their organizations. In this panel, we’ll dig into why the Alliance exists, the changes each of these companies are seeing in the landscape of product development, and what shifts are coming next.

You and your team want to experiment, but now what? Knowing the correct tactics and processes, you need to get started can feel like a massive mountain to climb. In this session, find out some of the tactics Sophie has adopted to grow Skyscanner’s experimentation and testing activities. These best practices will help you get started with taking your team from newbies of statistics to experimentation stars.

For many years, software engineering meant delivering software the way we constructed buildings. Over time, this belief turned out to be a wildly inefficient way to create innovative products and services based on software.

In this talk, I’ll discuss how modern approaches developed over the last twenty years - lean, agile, DevOps, and user experience design - enable teams to much more rapidly deliver software that delivers value to users.

I’ll also talk about how you can adopt these practices in the safety of your own office, and explore the art of the possible and the patterns and methods used by high performing organizations.