Inside the Mind of Lukas Vermeer: Shaping Experimentation Cultures at Vista

His focus extends beyond just running tests. At Vista (formerly Vistaprint), he’s building a culture where continuous learning drives progress. Despite the AI hype in experimentation, Lukas brings a practical perspective to where the field is heading. We took the time to discuss his views on the domain for Martech Family.

Lukas wrote his first program at age eight – a simple tool that combined random words to generate silly insults. This playful start with coding sparked a curiosity that would shape his entire career, growing from childhood experiments into a deep expertise in solving problems through code and data.

His path to experimentation really took off at Booking.com in 2013. Though he initially worked on recommendation systems, the company’s intense focus on A/B testing quickly pulled him into a new world. ‘Everything at Booking revolves around testing. It didn’t take long before I was completely immersed in it.’

‘At Booking.com, testing wasn’t just a process – it was part of the company’s DNA. If you had an idea, you didn’t spend weeks debating it—you just tested it,’ Lukas explains. ‘It was that simple.’ The company ran thousands of A/B tests, with everyone from product managers to engineers contributing to the experimentation pipeline.

What struck him most wasn’t just the technical setup, but how deeply testing was woven into the company’s mindset. ‘Booking had built an environment where experimentation was easy and fast. But more importantly, people were encouraged to try things, to fail, and to learn quickly from those failures.’

Booking.com is often cited as one of the world’s leading companies when it comes to scaling experimentation. The company runs thousands of A/B tests, with product managers, designers, and engineers all expected to contribute to the experimentation pipeline. This culture of experimentation permeates every level of the organization, and it’s where Lukas cut his teeth in the discipline.
“At Booking, experimentation was part of the fabric of the company,” Lukas explains. “If you had an idea, you didn’t spend weeks debating it—you just tested it. It was that simple. The focus was always on learning as fast as possible.” For Lukas, this approach aligned perfectly with his own philosophy. He thrived in the environment, quickly becoming a key figure in helping teams run more efficient and impactful experiments.

However, the real lesson Lukas took from his time at Booking was the importance of building strong foundations for experimentation—not just from a technical standpoint, but from an organizational one. “Booking had built an infrastructure where experimentation was easy and fast. But it wasn’t just about the tools. It was about the mindset. People were encouraged to try things, to fail, and to learn quickly from those failures.”

This culture of experimentation would later inform his approach when he joined Vista in 2020, where his mission was to not only improve the infrastructure for running experiments but also to shape the company’s organizational mindset.

Lukas’ transition to Vista presented a new challenge. Unlike Booking, where experimentation was already deeply embedded in the culture, Vista was at a much earlier stage in its journey. Here, Lukas found himself tasked with building both the technical and cultural foundations needed for large-scale experimentation.

“When I arrived at Vista, the biggest challenge wasn’t technical—it was organizational,” Lukas explains. “People didn’t all fully appreciate the value of experimentation or how it could be applied to their work. There was some skepticism.”

To address this, Lukas took a two-pronged approach. First, he focused on building a technical infrastructure that would allow teams to run experiments quickly and efficiently. Vista’s existing front-end architecture presented a unique challenge – while it was optimized for performance, it made it difficult to serve different versions of the site to different customers. “The infrastructure was too rigid,” he says. “This affected not just our ability to run experiments but also our personalization capabilities.”

Over the past year, Lukas has been leading efforts to overhaul this infrastructure, making it more adaptable and scalable. “Once we solve this infrastructure problem, I’m confident we’ll get back to doubling the number of experiments we run within the next year”. The goal is to create a system where teams can test ideas rapidly without being bogged down by technical limitations.

While improving the infrastructure was crucial, Lukas knew that the success of experimentation at Vista depended on more than just technology. To drive cultural change, he implemented several innovative initiatives.

One of his most creative approaches was the Ambassador Program. He personally sent signed copies of a book about online experimentation to each ambassador. “The postage to Australia was actually more expensive than the book itself – about $30 for shipping versus $20 for the book,” he recalls. “But it was worth it to create that personal connection and sense of commitment.”

However, the program faced challenges. “We had underestimated the amount of time ambassadors would need to spend on the program,” Lukas admits. “Since they didn’t report directly to me, their managers became concerned about the time investment versus the return they were seeing.”

To make experimentation more visible and accessible across the organization, Lukas also transformed how teams communicated about testing. He noticed that many questions about experimentation were being asked in private Slack threads, making it difficult for others to learn from these discussions. His solution was to create a public #help-experimentation channel, where he actively encouraged non-team members to answer questions and share their knowledge.

“When someone outside the team would help answer a question, I’d go out of my way to thank them publicly,” he explains. “It’s about making the good behavior visible and rewarding people for contributing, even when it’s not their direct responsibility.”

To track progress, his team implemented specific metrics around team penetration – measuring how many teams were actively running experiments. They also worked closely with the user research team to integrate experimentation into their discovery documentation, ensuring that testing became a natural part of the product development process.

“People resist change not because they don’t understand it, but because they can’t see how they fit into the new system,” Lukas notes. “My job is to help them see that experimentation is for everyone.”

While Lukas helped make significant progress in building Vista’s experimentation capabilities, he remains candid about the challenges that come with running experiments at scale in today’s digital landscape. “Web development is getting more complex, not less”.
This growing complexity, Lukas argues, presents a significant hurdle for companies that want to scale experimentation. “It’s not just about having the right tools,” he says. “It’s about the entire ecosystem—your infrastructure, your people, your processes. If any one of those things is out of sync, you’re going to hit bottlenecks.”

At Vista, Lukas is addressing these bottlenecks head-on. His work on the front-end infrastructure is aimed at removing many of the technical barriers that prevent teams from running experiments. “My team spent the last year making sure we have a system that can handle rapid experimentation at scale. Once that’s in place, we’ll be able to move much faster”.

Despite his enthusiasm for data-driven decision-making, he remains cautious about the promises of AI in experimentation. “I’m a bit of a pessimist when it comes to AI,” he admits. “AI can help optimize certain processes, but it’s not going to revolutionize experimentation the way some people think.”

For him, the real value of AI lies in its ability to augment human decision-making—not replace it. “AI can help you scale what you’re already doing, but it won’t tell you what to do next,” he explains. “It’s great for automating certain tasks, like personalizing recommendations, but it’s not going to come up with the next big idea. That still requires human intuition.”

Lukas is particularly skeptical of the idea that AI can fully automate experimentation. “AI can optimize experiments, but it can’t run them on its own,” he says. “You still need humans to set the parameters, interpret the results, and make decisions based on those results. AI is just a tool—it’s not the solution.”

As Lukas looks to the future, he remains optimistic about the potential for experimentation to transform organizations. However, he is quick to point out that the journey is far from over. “There’s always going to be the next bottleneck—whether it’s infrastructure, design resources, or organizational buy-in. Experimentation is a continuous process, not a one-time project”.

At Vista, Lukas’s focus for the next year is on scaling experimentation even further, both in terms of the number of tests run and the impact those tests have on the business. “I’m confident that once we fix the infrastructure bottlenecks, we’ll see an increase in the number of experiments we run, until we run into the next bottleneck. And then we’ll try to fix that one!” he says.

Lukas Vermeer’s journey—from writing his first lines of code as a child to shaping experimentation cultures at Booking.com and Vista—offers insights for companies aiming to build a culture of experimentation. Here are some of the key lessons:

1. Define Success Clearly: Before running any experiment, it’s crucial to define what success looks like. This clarity ensures that teams can measure progress accurately and make decisions based on solid data. “Without a clear definition of success,” he notes, “you’re just throwing darts in the dark.”

2. Build a Culture of Experimentation: Experimentation thrives in environments where testing and learning are celebrated. At Booking, the culture was designed to empower everyone—from engineers to marketers—to experiment and learn quickly. “The tools are important, but mindset is everything”.

3. Invest in Infrastructure: While mindset is key, fast and scalable experimentation also requires the right infrastructure. Without flexible systems, even the best ideas can be bottlenecked by technical limitations. Lukas’s work at Vista over the past year, focused on transforming the infrastructure to support rapid testing, is a testament to the importance of this foundation.

4. Empathy and Mentorship Matter: Leadership in experimentation is not just about driving technical change. It’s about guiding teams through the cultural shift, mentoring them through challenges, and celebrating wins—even when experiments don’t produce the desired results. “People resist change because they don’t see themselves in the new system. My job is to help them see how experimentation benefits them.”

5. AI is Just a Tool: While AI has the potential to optimize certain processes, Lukas remains sceptical of its role in fully automating experimentation. “AI can help scale what you’re doing, but it won’t tell you what to do next,” he says. For him, the human element—intuition, creativity, and judgement—is what drives true innovation.