Delivering Value from the Valley
September 20, 2012 9:00 AM PT

Ashok Chandra foresees a future in which the search engine has evolved to the point where it can engage in a discussion with you, anticipate your needs, and, through its knowledge of your preferences and history, become transformed into nothing less than an indispensable personal assistant.

“The key to successful collaboration between research and product groups,” says Chandra, a Microsoft distinguished scientist based at Microsoft Research Silicon Valley, “is that you need a vision of the future. We like to create ambitious goals and then try to achieve them.”

That’s the sort of thinking that has made Microsoft Research Silicon Valley a formidable presence in the world’s foremost hotbed of technological innovation, one whose distributed-computing focus has contributed technologies to such Microsoft products as Hotmail and Bing to Windows Azure and even Kinect. Now, with Microsoft Research’s 20th-anniversary year drawing to a close, let’s take a few minutes to acknowledge the remarkable ascent of this Silicon Valley institution over its initial decade.

Two Men, One Vision

From the outset, Microsoft Research Silicon Valley stemmed from the joint vision of two men, Roy Levin and Mike Schroeder, who provide the lab’s managerial responsibilities to this day.

Roy Levin
Roy Levin

“We shared a philosophy of how a lab should work, based on our years of working in labs managed by Bob Taylor,” recalls Levin, referring to Xerox’s legendary Palo Alto Research Center and DEC’s Systems Research Center.

The two helped to start the Systems Research Center, and 17 years later, when the opportunity to create a lab for Microsoft in Silicon Valley arose, they jumped at it.

“There was a perfect fit,” Levin says, “between Rick Rashid’s vision for research and everything we knew about building and running a research lab. The prospect of starting the lab was very, very exciting.”

Schroeder nods in agreement.

“The early days of a new lab are just filled with wide-open opportunity,” he adds. “One of the most important things is to get a new lab to a critical mass, combining researchers with varied yet overlapping interests to create a dynamic atmosphere for collaborative research. The focus of the first six months was to find that critical mass of people, get them on board, and make that happen as quickly as we could.”

Recruiting Theory

It didn’t hurt that Schroeder and Levin both came armed with expertise in computing systems. But they were aiming at something broader.

“One of the things we wanted to do was to have a substantial component of computing-theory research in the lab,” Levin says. “That made it important to get some top people who were well-known in the computing-theory community and would make others sit up and take notice that we were establishing a lab in the area.”

That mission succeeded early on, with the hiring of Cynthia Dwork, Leslie Lamport, and Andrew Goldberg, leading theorists in technical areas of particular relevance to the new lab.

“We were able to get some very well-known, world-class people on board early,” Schroeder confirms, “and that put us on the theory map.”

‘Delicate Undertaking’

Ted Wobber, too, remembers the lab’s early days, when he was one of the first hires.

Ted Wobber
Ted Wobber

“Starting a research lab is a fairly delicate undertaking,” he says. “It takes a fair amount of time and effort to develop the momentum you need to do good research. From the very beginning, we tried to branch out and hire younger people and people with different backgrounds into this organization, and I think we succeeded pretty well.”

Wobber’s early efforts at Microsoft Research Silicon Valley focused on operating systems and security. These days, he’s applying his skills to security public-key infrastructures and to CORFU, a distributed storage architecture using flash memory.

“Those are very different beasts,” he says. “One of the really good aspects of this lab is that you can work on things that are in widely distributed areas. The management of the lab encourages people to have an open mind, to be skilled in various areas, and to be prepared to collaborate with other people who are working in very different areas.”

The Word Is Flat

That encouragement is fostered in part by the lab’s flat managerial structure, one of the benefits that Levin and Schroeder experienced in Taylor’s research labs and about which Levin has published an influential paper.

“You want to have the best people you can have working collaboratively in a lot of different areas with a minimum of managerial overhead,” Wobber says. “There’s an underlying relationship going on here. To have a flat organization, the manager has to trust you to spend your time reasonably. It’s your responsibility to live up to the expectations of your manager to do that. There’s a nice bargain that takes place, and I don’t know many working environments in which that bargain exists.”

Personnel at Microsoft Research Silicon Valley
Researchers at Microsoft Research Silicon Valley: 10 years on, still going strong.
The guiding principles produced strong results and, consequently, have influenced the management style elsewhere within Microsoft Research.

“From our experiences in previous labs, we saw value in the kinds of collaborations that spring up when you don’t have artificial group boundaries, which constrain people to look to other people in the group as their collaborators and not elsewhere,” Levin says. “The lab has worked without subgroups since its inception, and we’ve gotten some quite excellent work out of that.”

Sometimes, Schroeder says, such collaborations can lead to pleasant surprises.

“If you get the machine working in a healthy way,” he observes, “it becomes kind of an ecosystem, and it naturally spawns interesting and unexpected projects. That’s the part that I find most rewarding. It happens more often than you might realize that some people have gotten together over coffee and hatched an idea, and they just went off and did it.

“Sometimes, there are really surprising bedfellows, and it leads to really amazing results.”

Hardware, Software, and Security

Take Fang Yu, for example. Since joining the lab in August 2006, she has worked primarily in network security. But while earning her Ph.D. at the University of California, Berkeley, she wrote her thesis on hardware techniques for improved security. Now, in addition to her ongoing work to thwart network attacks, she also is exploring hardware solutions to avoid packet loss, and one of her colleagues in this effort is none other than Chuck Thacker, recipient of the 2009 A.M. Turing Award, the most prestigious honor in computer science.

“I’m a believer,” she says, “that specialized hardware can help computation. Not everything needs to be software.”

Yinglian Xie and Fang Yu
Yinglian Xie (left) and Fang Yu

Yu has worked closely over the last six years with Yinglian Xie, who joined Microsoft Research Silicon Valley from Carnegie Mellon University in the same month as Yu. Together, they have pursued game-changing research to make it more difficult for Internet attacks to succeed. Their analysis of IP address properties provides a wealth of information about attackers. They analyze user populations to distinguish between benign and malicious users. And they have examined malicious infrastructures such as botnets to determine how to deflect attacks conveyed via online advertising.

“By looking at the interactions from the infrastructure perspective,” Xie says, “we’re trying to tackle the fundamental resources needed by attackers. Hopefully, it can change significantly the economics of attackers.”

Now, Yu and Xie, acting in concert with product groups, are combining those approaches, with the goal of providing an even more robust approach to protecting Internet users.

“We are building a unified infrastructure to help combat many of these attacks,” Yu says. “That’s very exciting and will be a bigger accomplishment.”

Success and Impact

Working at Microsoft gives the researchers a significant advantage.

“I work with product groups,” Xie says, “and that kind of opportunity can potentially generate impact that wouldn’t be possible in a purely school environment.”

Yu also appreciates the boost her research gets from access to Microsoft’s broad user base.

“I’m not only doing something that has immediate impact,” Yu says, “but I also have impact because Microsoft software serves so many people. We are developing techniques that help millions of people around the world.”

Such results stem directly from the way the lab operates.

“The two major ways in which we have impact,” Levin says, “are on the academic world, through publications, and on the company, through interactions with the product groups in which they take ideas from research and embody them in products.”

Microsoft Research Silicon Valley researchers appreciate that two-pronged approach.

“My biggest measure of success is that we have succeeded in recruiting serious, world-class research talent,” Schroeder states, “and when you talk to these people heart-to-heart, they love it here. This is an environment where they can be really productive and happy and collaborate in a way where they don’t have to worry about protecting their own ideas, their reputations, or their personal contributions.”

Accelerating Progress

That brings us back to Ashok Chandra. He says that working at Microsoft Research—and its Silicon Valley presence, in particular—is a rare and precious opportunity.

Ashok Chandra
Ashok Chandra

“I like to think about exciting technologies for the future and make big things happen,” he says. “Microsoft Research has enormous talent, and the culture gives you the freedom to pursue your dreams. That combination is just terrific.”

In his case, he gets to pursue the dream of extending the capabilities for search far beyond the delivery of 10 blue links.

“We had a rather grand vision,” he says. “Search is the only software system that you can talk to about any topic at all. You can ask a search engine, ‘What’s the weather like in Seattle?’ You can’t do that in a database system or DOS or Excel. That combination is incredibly powerful, and I thought that’s the way the future of human-computer interaction would unfold.”

That grand vision has two corollaries. One is that search could evolve into a task engine that helps people do things. The other is that this kind of interaction could be extended to all Microsoft software, be it Office or Windows or Xbox or Skype.

“It’s not just a transfer of technology,” Chandra says. “it’s a whole vision of how we are going to accelerate progress. That’s our role in Microsoft Research, after all.”

Intent, Entities, and Apps

The first step in bringing that vision to reality was to enable the search engine to identify user intent. By enabling the search engine to remember some information about the user, it is better able to provide results that are more relevant.

Today, if you search for “flight status” on Bing, it will ask for your airline and flight number, and once you respond, it will deliver pertinent details. It seems like an obvious solution, but many different layers of the service suddenly had to be equipped to process the state of a query. Data about all the flights in the world had to be gathered, processed, and distributed every minute. The distribution system had to be constructed, and the user-interaction paradigm had to change.

The next step was to enable an action to help users with tasks. Actions apply to entities, Chandra and colleagues deduced, such as “I want to watch a movie.” Again, simple concept, challenging execution. Movies can have similar titles, necessitating a process of disambiguation. It’s not as easy as saying, “I want to watch Avatar.” Which Avatar? There are other films with the same title.

As it turns out, quite a few Microsoft product teams also were seeking ways to harness entity knowledge. What Chandra and his collaborators built became a product platform available to engineers from across the globe.

In addition to identifying the state of a query and the entities upon which action can be taken, the platform also has to work with other applications. It’s not enough to find the right Avatar, you also need to retrieve it from a service such as Netflix—and you might want to check your friends’ opinions about the movie on Facebook.

The combination, though, can be powerful. It is possible, Chandra says, that rather than a search engine being involved in one or two interactions with an individual, it eventually could engage in an actual conversation with the user, asking increasingly detailed questions to determine the precise action desired.

“In the future, it will be an organic process between you and the computer or device,” Chandra says. “They might be connected to the cloud, or to several clouds, but one could imagine the dialogue progressing in a way that could have a whole community engaged in that dialogue over time.

“Privacy is going to very important, so we need to be extremely thoughtful, but if the user permits the system to remember the historical part of a dialogue, the system can be that much more helpful in suggesting things—even, perhaps, before the user thinks about them.”

In a way, that’s the charter of Microsoft Research Silicon Valley: to envision the future and to invent the tools that will enable it to come into existence.

Community and Commitment

Not only does the lab contribute extensively to the Microsoft and academic communities, it also demonstrates a strong commitment to its Silicon Valley surroundings, both through efforts with local universities and directly with neighboring organizations.

As part of its anniversary activities, the lab is working with the Boys and Girls Club of Menlo Park to provide funding for a new audio-visual studio to supplement the club’s teen program.

“We are contributing technology as well as funds for construction and staffing of the studio,” Levin says. “Our goal is to involve youth with technology like Kinect, giving them ways to experiment with things like gestural game development, which we hope will keep them excited and engaged with technology at a critical stage of their development. After all, that’s where the next generation of researchers will come from.”

Mike Schroeder
Mike Schroeder

Schroeder takes a moment to reflect on the current generation of researchers at Microsoft Research Silicon Valley.

“I am very happy,” he says, “that, after 10 years, we have amassed an amazing mix of people with brilliant ideas that they can act on quickly, along with a pipeline of basic, long-term, transformative things going on as time progresses.”

Expect that to continue.

“I don’t see the mission of the lab changing dramatically from what it’s been for the past 10 years,” Levin concludes. “It’s not like there’s a dramatic turn ahead. The lab is at its optimal size right now—it’s as big as it can be with the flat structure that we have and still be manageable. The technical focus will not change abruptly, but as the field evolves, we will continue to look at where the action is and is likely to be, and the people who flow through here will reflect that.

“The future of the lab is, at one level, more of the same, but more of the same means more innovation. Innovation is what we do, but how it gets done, naturally, evolves to respond to the environment and the times.”