Testing cookie behavior across hundreds of web surfaces with our in-house auditor

As AI and automation become a bigger part of digital experiences, it’s increasingly important that people stay in control of the choices they make. They should be able to benefit from new technology without losing the preferences they’ve already expressed.

Cookie banners are one of the most familiar places where people express those preferences online. They’re also one of the most visible parts of a company’s privacy program. And for companies that operate a large, dynamic collection of websites, keeping them working as intended can be challenging. A banner might work correctly when a page launches, but changes made later can affect how someone’s cookie preferences are saved or applied. When someone accepts, declines, or customizes cookies, the website needs to remember that choice and behave accordingly. New integrations, experiments, or even changes to a page’s settings can unintentionally affect that behavior.

Manually checking that cookie banners continue to work as expected across all of our websites requires hours of quality assurance. To make those checks more manageable, we built a cookie auditor that acts like a privacy-conscious user. It visits Dropbox web pages and checks that they only load cookies consistent with that user’s privacy preferences. This gives us a scalable way to put our privacy commitments into practice as our web presence continues to evolve.

Most of us are familiar with cookies, but we don’t always think about what they mean for the choices we make online. Cookies are small pieces of data sent to a browser when someone visits a website. Some are strictly necessary, meaning they serve an essential purpose like keeping a user logged in or remembering language settings. Others are non-essential and support things like analytics, marketing, performance, or advertising. Strictly necessary cookies load automatically, but privacy laws in some places require companies to get consent before loading non-essential ones.

On the surface, cookie compliance can seem pretty straightforward. Websites show a banner, provide buttons to accept or decline, record the choice, and load cookies accordingly. But at scale, things can get a bit more complicated. Dropbox operates over 200 web surfaces across different products and teams, and the cookies on each page can vary depending on that surface’s purpose. URLs also change all the time as pages are launched, retired, redirected, localized, or included in experiments.

Plus, cookie banners aren’t the only way people can communicate their privacy preferences. Global Privacy Control (GPC), for example, is a technical standard that lets people set a privacy preference in their browser that can be communicated automatically to the websites they visit. Our system needed to recognize GPC signals and respect those preferences, too.

It’s also important to look beyond cookie banners themselves. Even when a banner appears to work correctly, we still need to verify that someone’s choice is actually being applied. That means checking which cookies load before and after someone makes a selection, and that their preference continues to apply after the page reloads. That’s where our cookie auditor comes in.

Before we began building a cookie auditor, our Privacy and Engineering teams first needed to agree on what the system should consider correct behavior. That meant deciding which websites should be audited, what kind of banner behavior would constitute a violation, and which third-party cookies and other artifacts should be tracked separately. We also defined when a missing banner should be considered a problem and when it was expected, such as on a web page that was no longer in use.

Privacy programs often depend on legal concepts that require human interpretation, such as opt-in or opt-out, strictly necessary, and affirmative consent. But a machine needs much more concrete instructions about what should happen when someone makes a particular choice and what to flag when something doesn’t behave as expected. Turning those concepts into outcomes the cookie auditor could actually test was an important part of the work.

We also kept these classifications outside the auditor’s source code. That meant the Privacy team could update the list of approved cookies and known exceptions without waiting for engineers to change the code and release a new version. Importantly, it also made the system easier to adapt as Dropbox services and regulatory guidance evolve, helping our privacy protections keep pace with both.

Because we built our cookie banner in-house rather than using an off-the-shelf product, we could integrate the auditor directly with our existing consent infrastructure. That gave us more control over how the auditor and consent system worked together and made it easier to adjust what we tested as our websites, services, and privacy requirements evolved. Instead of treating compliance as a fixed set of rules, we built a system that could change along with those requirements.

Defining the criteria that’d govern our cookie auditor was only one part of the challenge. We also needed a way to test those criteria against what someone really experiences when they visit a Dropbox website. To do that, we designed the auditor to behave much like a real visitor would. It uses Playwright, a browser automation library, to open a Dropbox web page in a fresh, isolated browser session and observe what happens from the moment the page loads.

For every page we check, the cookie auditor runs three separate tests to account for different privacy experiences. One simulates a standard US visitor, another simulates a visitor in the EU, and the third uses a GPC signal. Each test starts from scratch, with no existing cookies or saved preferences, so the auditor sees the same experience as a new visitor. Before interacting with anything on the page, the auditor records which cookies have already loaded and checks whether they match what we expect to see for that test. 

Next, the auditor finds the page’s consent controls and declines non-essential cookies. That’s more complicated than simply looking for a button that says “Decline.” Depending on the page, consent controls might appear in a banner, a floating control, a preferences window, or a footer link. They can also appear in any of our 22 supported languages. Rather than relying on specific words on the screen, the auditor identifies and interacts with the underlying consent controls. It then reloads the page and checks what happens next.

Reloading the page is an important part of the test. Once the page reloads, the auditor checks that the user’s preference still applies and that the cookies present match what we expect for that test. If anything unexpected appears, it’s recorded for further review. This focus on what actually happens, rather than what should happen based on a configuration, became one of the project’s most important design decisions. 

At this point, we had a reliable way to test whether a Dropbox web page was respecting someone’s privacy choices. But the auditor could only test the pages we knew about. As new pages were created and existing ones changed, we also needed a reliable way to identify which pages should be included in our audits.

Although the auditor worked well on our defined list of URLs, we soon realized that keeping that list up to date would be an ongoing challenge. New marketing pages, blog posts, and help center articles are published every week. Our Privacy team didn’t necessarily have clarity on every new web surface being created across the company, and asking teams to manually send us every new URL would create exactly the kind of bottleneck the project was meant to reduce.

Our solution was a companion URL detector, which is effectively an auditor for our auditor. The cookie auditor checks whether a user’s decision is respected, and the URL detector makes sure we’re checking all the places where the consent experience should appear.

We started by using traffic data to identify Dropbox web properties. Because that data includes a new record every time someone visits a page, our initial search had to comb through billions of records. But billions of visits didn’t mean billions of different pages. The same URL path might appear thousands or even millions of times, so we first filtered out repeated records to narrow the results down to a more manageable list of unique paths.

From there, we applied a second, more detailed round of filtering. Doing all of that work while searching through billions of traffic records would have required significantly more time and computing resources, but working from the smaller list made it much more efficient. At this stage, we could exclude pages that didn’t need to be tested, group together pages that used the same consent logic, and select representative URLs from large groups of similar pages. This gave the cookie auditor a manageable set of pages that still represented the different consent experiences someone might encounter across Dropbox.

Automation can narrow the search space, gather evidence, and identify potential issues, while Privacy and Engineering teams provide the insight needed to understand unusual pages and exceptions. This division of labor helps us keep pace with Dropbox’s changing web footprint without creating another time-consuming manual process for our teams. It also reflects a principle behind how we approach automation and AI at Dropbox: Use these tools to surface relevant information so that people have the context they need to make informed decisions and take action, while spending less time searching for and sorting through it.

Privacy programs rely on policies, controls, and review processes, but people ultimately experience them through the products they use. When someone makes a choice about their data, they expect the product to respect it. That means making the choice clear, making declining as easy as accepting, remembering the preference, and applying it consistently as someone moves across a website.

That’s why we think about privacy much like reliability or security. It requires ongoing attention as products and systems change. Building the auditor showed us just how many pieces needed to work together to make that possible. We needed to know which pages to test, distinguish real issues from false positives, translate legal requirements into behavior that the auditor could check, and test the experience from a user’s perspective rather than relying on how a system was configured.

Over time, that changed how we thought about the auditor itself. It gave us a way to turn our privacy commitments into something we could regularly test against the actual experience people have on our websites. For us, building that trust into a changing product means making clear commitments to users and designing systems that continuously verify we’re upholding them.

Today, the auditor gives our Privacy and Engineering teams a weekly report on consent behavior across Dropbox websites. It separates likely violations from known false positives, making it easier for teams to understand what needs their attention. Historical results also help us track trends and spot when something that was working correctly begins to change. Our next step is to make that process even more efficient by connecting findings to the teams responsible for those pages and getting potential issues to the right people more quickly.

For teams considering a similar project, the browser automation itself may be the simplest part. Much of the work happens around ‌the browser automation, including defining what correct behavior looks like, maintaining a reliable inventory of what needs to be tested, separating real issues from noise, and creating a review process that brings the right teams together. Investing in those pieces has given us greater confidence that our cookie program continues to work as intended, even as the websites and systems around it change.

Acknowledgments: Special thanks to folks on the Privacy Engineering and Legal teams for guidance throughout the project.

~ ~ ~

If building innovative products, experiences, and infrastructure excites you, come build the future with us! Visit dropbox.jobs to see our open roles.