AI news

Which metrics matter for personal agents? From app opens to delegated work and completion

By 赛博禅心1 viewsSource: 赛博禅心
Personal agent metrics, delegated work and task completion

Saibo Chanxin traces the evolution of internet metrics to discuss personal agents’ delegation volume, completion rates and revenue, while noting that definitions of delegated work still need to be aligned.

Ant’s Lingguang has released figures for the first half-month after the launch of version 2.0: 3 million matters delegated to it and 11 million task executions (one week after launch, those two figures were 1.16 million and 3.6 million).

Delegation volume may currently be one of the more reasonable North Star metrics for a personal agent.

But! This article is not about Lingguang—that was a feint. Instead, following the metric of “how many matters have been delegated,” I want to discuss: What metrics should personal agents use?

For background, over the past 20 years, whenever we have evaluated an internet product or project, we have always needed a reference metric. For example:

• In the web portal era, we looked at PV: every additional page opening meant another chance to sell advertising, so screens filled with pop-ups (like the one beside this text).

• In the social media or mobile internet era, we looked at daily active users.

• For video sites, we looked at total time spent.

These are what we call North Star metrics.

Advertisement | Fantasy martial-arts browser game | Close ad | The Legend of Mir | Close ad | System notification | Close | VIP membership center | Too many ads? Become a member for ad-free reading across the site | First month ¥9.9 | Original price ¥30 | Subscribe now | Close

But in the age of agents—or, more specifically, the current age of personal agents—what should the North Star metric be? There is, in fact, no settled answer yet.

Where North Star metrics come from

A North Star metric is the single number a project chooses as the best representation of its progress. Typical examples include DAU, GMV and ARR.

Article illustration

We have been through several waves of the internet, and every new wave of product formats brings new metrics. At the very beginning, for instance, how did we judge how impressive a website was? We used its hit count: the number of interactions between users and the site.

For example, if a website had dozens of images, opening it once would send dozens of requests to the server. So the first generation of websites looked something like this:

Article illustration

Hmm… Back then, someone even devised a full expansion of HITS: How Idiots Track Success (a way for idiots to measure success).

Later, people switched to counting pages: one opening counted as one view, or PV. PV remained in use for many years, until Ajax became popular—Gmail and Google Maps both used it. New content could load on its own without refreshing the page. You could spend half an hour in Gmail and still register only one PV.

In 2007, Nielsen simply switched to ranking websites by time spent. The company had started out measuring television ratings, and US advertisers allocated their money according to its rankings. The result? AOL (America Online), which had ranked only sixth by PV, suddenly became number one in the United States. After all, many Americans at the time used AOL for email and kept its chat services running.

From hits to PV to time spent, every change in counting method happened because the old number was rising without advertisers’ money buying any more user attention.

Metrics determine what a product looks like

No product operates as a single, unified machine; it is made up of people and departments. So when those above set performance targets for those below, the requirement is to optimize a particular number to a particular level.

The project then becomes an effort by the entire company to optimize one or several numbers, with monetization designed around those numbers as well. These two forces influence each other, and in the end everyone works extremely hard to optimize them.

Take a web portal: it earns money by selling advertising space, charging a certain amount per thousand impressions. But there are only so many ads that can fit into any one display or page layout. So people at the time did everything they could to increase exposure for the same piece of information, carrying it across sites and onto more pages.

Think back to websites around 2005. Didn’t they present the extraordinary sight below?

Article illustration

But clearly, damaging the user experience this way would make more people choose competing products. As a footnote, around 2010 the US gossip blog Gawker found that one post on its network had received 14,308 PV but brought in only 756 new readers. Gawker therefore changed editors’ bonuses to be calculated by unique visitors (UV).

Now consider social networks.

Looking back, that generation of products was competing for people’s attention, and one of the best measures of that was DAU.

When Facebook went public in 2012, it had 483 million daily active users. After more than a decade of working around that number, by the end of 2023 its daily active users had risen to 2.11 billion.

China Mobile | Thursday, November 12 | 9:41 | Your Agent: If you don’t open me now, my performance pay will be docked | Your Agent · 5 minutes ago: You haven’t opened me for 2 days | Your Agent · 21 minutes ago: Your friends are using me too. Come see what they’ve been asking | Your Agent · 1 hour ago: You might want to ask: What’s the weather like today? | Your Agent · 2 hours ago: Long time no see. I have 3 little things to talk to you about | Your Agent · 3 hours ago: Check in today for 10 points; check in for 7 days in a row to get membership

Article illustration

Video sites, meanwhile, focused on time spent. In its early years, YouTube recommended videos by view count, with clickbait making up much of what rose to the top. In 2012, YouTube switched to recommendations based on watch time and set a target: increase daily watch time from 100 million hours to 1 billion hours, a target it reached four years later.

Video Site | Search for what you want to watch | Search | Upload | Log in | 05:00 | HD | Shocking! He changed just one metric and daily active users grew tenfold | Traffic Formula Research Institute | Subscribe | 1.28 million | 98,765,432 views | 3.18 million | 24,000 | Related videos | Autoplay | 00:15 | A cat watched this video and froze for a full three seconds | Cat Caretaker Daily | 12.03 million views | Up next | 00:15 | A cat watched this video and froze for a full three seconds | Cancel | 05:00

At this point, you can see that the main revenue source in this wave of the internet was advertising, which required selling users’ time to advertisers. That is why we had metrics such as PV, daily active users and time spent to compete for users’ attention. These metrics and product formats also shaped one another: the metric you choose determines the kind of product you ultimately get.

Personal Agent

Now that we are in the era of personal agents, what metrics should we look at? The first thing to rule out is time spent, because personal agents do not generate revenue by selling attention.

Personal agents earn money from users paying to “get things done”: either through monthly personal subscriptions—individual memberships for ChatGPT and Claude both start at US$20 a month—or, on the enterprise side, where charging for outcomes has already begun.

As for interactions and time spent, they all burn through costly tokens, so the less of them, the better. The better a personal agent gets things done, the fewer times a user needs to open it.

Article illustration

This sort of thing, where the data runs counter to intuition, has happened before.

During World War II, statisticians in the United States helped the military analyze bullet holes in returning bombers. The wings and fuselage were densely covered, while the engines had very few. Intuition suggested adding armor where there were more bullet holes. Statistician Abraham Wald, however, recommended adding armor where there were fewer: after all, aircraft whose engines had taken a few hits had never made it back.

Article illustration

There is also a video site everyone knows, the “P site”—I won’t spell it out here. A story has long circulated that, in its early years, it judged whether a video was good by how many people closed it halfway through…

Hmm… because by then, the user had already “finished.”

GitHub | Search or jump to… / | 12:00 | HD | Experienced technician comes to your home to fix a Pipeline (full version) | ★ 12.4k · 12.04 million views · #hotfix | 98% LGTM | Code Review · Real-time assessment | User has left | Closed it at 52% | LGTM: A good video

If you still evaluate daily active users at this point, the team will send more push notifications and set more reminders, finding all sorts of ways to bring you back several times a day. That runs counter to “saving you the trouble.”

Conversely, a user who delegates something and never opens the app again is called an inactive user in a daily-active-user report, yet that user is getting the best use out of it.

Daily active user report / User details | Today’s daily active users, DAU: 1,024,768 | 3.2% compared with last week | Users’ app-opening status over the past 7 days | User 1024: Opened 12 times today · Active | User 2048: Opened 7 times today · Active | User 0429: Has not opened it for 37 days | In those 37 days, the things he delegated to the Agent: Watch for airfare price drops; cancel forgotten automatic renewals; pay water, electricity and gas bills; remind him of his child’s vaccinations; book a vehicle inspection… 12 matters in total, all completed | User 0001: Opened 3 times today · Active | The inactive user is getting the best use out of it

Let me make a joke first and introduce one of the most familiar personal agents: WeChat.

Article illustration

Supports multimodal input; response speed depends on its mood.

It contains a great many character agents. Some are hilarious, while others can solve all sorts of problems in your life. All these agents can read images, and can even make voice calls to you.

Article illustration

Allen Zhang’s idea for WeChat was “leave when you’re done.” Applied to a personal agent, it should be that you can leave before it is done.

What personal agents should measure

Following the reasoning above, people do not need to keep staring at a personal agent’s screen. If we really were to design a North Star metric for personal agents, what would it have to relate to?

  1. How many things the user has asked it to do.
  2. How many of those it has completed.

These two numbers also need to be linked to revenue. For example, customer service software company Intercom’s customer service agent charges directly for outcomes: for each customer issue it resolves, it charges US$0.99. (I haven’t actually found a very good example; I am just using it here as an illustration.)

Article illustration

Lingguang, mentioned at the beginning of this article, is probably moving in this direction too: competing for users’ authority to delegate matters. For example, you can ask it to watch whether a flight you have just booked drops in price, or to dig out automatic renewals you forgot to cancel… Once a condition is met, it will remind you, and for some matters it will take action directly. However many such matters you hand over is your delegation volume.

Article illustration

Of course, we may also see other metrics of this sort later:

  • Delegation rate: how many users are delegating matters.
  • Completion rate: what proportion of the matters handed over have been completed.
  • Retention measured by matter: 30 days after a matter is delegated, is it still in the AI’s hands?

Of course, nobody can say for sure what future metrics will look like. Even delegation volume itself still has a whole set of definitions to align. For instance… does setting an alarm count as delegating a matter? (Xiao Ai is overjoyed.)

Article illustration

This kind of personal agent is actually quite like autonomous driving. I previously read material saying that companies conducting road tests in California need to report to the Department of Motor Vehicles how far they drive on average before a human has to take over. Something like this could also be adapted to personal agents, although it might not be called a business metric; it might instead be a metric for optimizing product performance.

Finally

Perhaps another hot take will emerge later: the classic personal-agent user profile is someone who opens it frequently but spends very little time on each visit, while the machines responsible for handling tasks behind the scenes run continuously, 24 hours a day, packed with all sorts of tasks.

This article comes from the WeChat public account “Saibo Chanxin,” by Saibo Chanxin.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)