The National Security Agency is harvesting huge numbers of
images of people from communications that it intercepts through its global
surveillance operations for use in sophisticated facial recognition programs,
according to top-secret documents.
The spy agency’s reliance on facial recognition technology has grown
significantly over the last four years as the agency has turned to new software
to exploit the flood of images included in emails, text messages, social media,
videoconferences and other communications, the N.S.A. documents reveal. Agency
officials believe that technological advances could revolutionize the way that
the N.S.A. finds intelligence targets around the world, the documents show. The
agency’s ambitions for this highly sensitive ability and the scale of its effort
have not previously been disclosed.
The agency intercepts “millions of images per day” — including about 55,000
“facial recognition quality images” — which translate into “tremendous untapped
potential,” according to 2011 documents obtained from the former agency
contractor Edward J. Snowden. While once focused on written and oral
communications, the N.S.A. now considers facial images, fingerprints and other
identifiers just as important to its mission of tracking suspected terrorists
and other intelligence targets, the documents show.
“It’s not just the traditional communications we’re after: It’s taking a
full-arsenal approach that digitally exploits the clues a target leaves behind
in their regular activities on the net to compile biographic and biometric
information” that can help “implement precision targeting,” noted a 2010
document.
One N.S.A. PowerPoint presentation from 2011, for example, displays several
photographs of an unidentified man — sometimes bearded, other times clean-shaven
— in different settings, along with more than two dozen data points about him.
These include whether he was on the Transportation Security Administration
no-fly list, his passport and visa status, known associates or suspected
terrorist ties, and comments made about him by informants to American
intelligence agencies.
It is not clear how many people around the world, and how many Americans, might
have been caught up in the effort. Neither federal privacy laws nor the nation’s
surveillance laws provide specific protections for facial images. Given the
N.S.A.’s foreign intelligence mission, much of the imagery would involve people
overseas whose data was scooped up through cable taps, Internet hubs and
satellite transmissions.
Because the agency considers images a form of communications content, the N.S.A.
would be required to get court approval for imagery of Americans collected
through its surveillance programs, just as it must to read their emails or
eavesdrop on their phone conversations, according to an N.S.A. spokeswoman.
Cross-border communications in which an American might be emailing or texting an
image to someone targeted by the agency overseas could be excepted.
Civil-liberties advocates and other critics are concerned that the power of the
improving technology, used by government and industry, could erode privacy.
“Facial recognition can be very invasive,” said Alessandro Acquisti, a
researcher on facial recognition technology at Carnegie Mellon University.
“There are still technical limitations on it, but the computational power keeps
growing, and the databases keep growing, and the algorithms keep improving.”
Continue reading the main story
State and local law enforcement agencies are relying on a wide range of
databases of facial imagery, including driver’s licenses and Facebook, to
identify suspects. The F.B.I. is developing what it calls its “next generation
identification” project to combine its automated fingerprint identification
system with facial imagery and other biometric data.
The State Department has what several outside experts say could be the largest
facial imagery database in the federal government, storing hundreds of millions
of photographs of American passport holders and foreign visa applicants. And the
Department of Homeland Security is funding pilot projects at police departments
around the country to match suspects against faces in a crowd.
The N.S.A., though, is unique in its ability to match images with huge troves of
private communications.
“We would not be doing our job if we didn’t seek ways to continuously improve
the precision of signals intelligence activities — aiming to counteract the
efforts of valid foreign intelligence targets to disguise themselves or conceal
plans to harm the United States and its allies,” said Vanee M. Vines, the agency
spokeswoman.
She added that the N.S.A. did not have access to photographs in state databases
of driver’s licenses or to passport photos of Americans, while declining to say
whether the agency had access to the State Department database of photos of
foreign visa applicants. She also declined to say whether the N.S.A. collected
facial imagery of Americans from Facebook and other social media through means
other than communications intercepts.
“The government and the private sector are both investing billions of dollars
into face recognition” research and development, said Jennifer Lynch, a lawyer
and expert on facial recognition and privacy at the Electronic Frontier
Foundation in San Francisco. “The government leads the way in developing huge
face recognition databases, while the private sector leads in accurately
identifying people under challenging conditions.”
Ms. Lynch said a handful of recent court decisions could lead to new
constitutional protections for the privacy of sensitive face recognition data.
But she added that the law was still unclear and that Washington was operating
largely in a legal vacuum.
Laura Donohue, the director of the Center on National Security and the Law at
Georgetown Law School, agreed. “There are very few limits on this,” she said.
Congress has largely ignored the issue. “Unfortunately, our privacy laws provide
no express protections for facial recognition data,” said Senator Al Franken,
Democrat of Minnesota, in a letter in December to the head of the National
Telecommunications and Information Administration, which is now studying
possible standards for commercial, but not governmental, use.
Facial recognition technology can still be a clumsy tool. It has difficulty
matching low-resolution images, and photographs of people’s faces taken from the
side or angles can be impossible to match against mug shots or other head-on
photographs.
Dalila B. Megherbi, an expert on facial recognition technology at the University
of Massachusetts at Lowell, explained that “when pictures come in different
angles, different resolutions, that all affects the facial recognition
algorithms in the software.”
That can lead to errors, the documents show. A 2011 PowerPoint showed one
example when Tundra Freeze, the N.S.A.’s main in-house facial recognition
program, was asked to identify photos matching the image of a bearded young man
with dark hair. The document says the program returned 42 results, and displays
several that were obviously false hits, including one of a middle-age man.
Similarly, another 2011 N.S.A. document reported that a facial recognition
system was queried with a photograph of Osama bin Laden. Among the search
results were photos of four other bearded men with only slight resemblances to
Bin Laden.
But the technology is powerful. One 2011 PowerPoint showed how the software
matched a bald young man, shown posing with another man in front of a water
park, with another photo where he has a full head of hair, wears different
clothes and is at a different location.
It is not clear how many images the agency has acquired. The N.S.A. does not
collect facial imagery through its bulk metadata collection programs, including
that involving Americans’ domestic phone records, authorized under Section 215
of the Patriot Act, according to Ms. Vines.
The N.S.A. has accelerated its use of facial recognition technology under the
Obama administration, the documents show, intensifying its efforts after two
intended attacks on Americans that jarred the White House. The first was the
case of the so-called underwear bomber, in which Umar Farouk Abdulmutallab, a
Nigerian, tried to trigger a bomb hidden in his underwear while flying to
Detroit on Christmas in 2009. Just a few months later, in May 2010, Faisal
Shahzad, a Pakistani-American, attempted a car bombing in Times Square.
The agency’s use of facial recognition technology goes far beyond one program
previously reported by The Guardian, which disclosed that the N.S.A. and its
British counterpart, General Communications Headquarters, have jointly
intercepted webcam images, including sexually explicit material, from Yahoo
users.
The N.S.A. achieved a technical breakthrough in 2010 when analysts first matched
images collected separately in two databases — one in a huge N.S.A. database
code-named Pinwale, and another in the government’s main terrorist watch list
database, known as Tide — according to N.S.A. documents. That ability to
cross-reference images has led to an explosion of analytical uses inside the
agency. The agency has created teams of “identity intelligence” analysts who
work to combine the facial images with other records about individuals to
develop comprehensive portraits of intelligence targets.
The agency has developed sophisticated ways to integrate facial recognition
programs with a wide range of other databases. It intercepts video
teleconferences to obtain facial imagery, gathers airline passenger data and
collects photographs from national identity card databases created by foreign
countries, the documents show. They also note that the N.S.A. was attempting to
gain access to such databases in Pakistan, Saudi Arabia and Iran.
The documents suggest that the agency has considered getting access to iris
scans through its phone and email surveillance programs. But asked whether the
agency is now doing so, officials declined to comment. The documents also
indicate that the N.S.A. collects iris scans of foreigners through other means.
In addition, the agency was working with the C.I.A. and the State Department on
a program called Pisces, collecting biometric data on border crossings from a
wide range of countries.
One of the N.S.A.’s broadest efforts to obtain facial images is a program called
Wellspring, which strips out images from emails and other communications, and
displays those that might contain passport images. In addition to in-house
programs, the N.S.A. relies in part on commercially available facial recognition
technology, including from PittPatt, a small company owned by Google, the
documents show.
The N.S.A. can now compare spy satellite photographs with intercepted personal
photographs taken outdoors to determine the location. One document shows what
appear to be vacation photographs of several men standing near a small
waterfront dock in 2011. It matches their surroundings to a spy satellite image
of the same dock taken about the same time, located at what the document
describes as a militant training facility in Pakistan.
A version of this article appears in print
on June 1, 2014,
on page A1 of the New York edition with the headline:
N.S.A. Collecting Millions of Faces From Web Images.
Joseph J. Atick cased the floor of the Ronald Reagan Building
and International Trade Center in Washington as if he owned the place. In a way,
he did. He was one of the organizers of the event, a conference and trade show
for the biometrics security industry. Perhaps more to the point, a number of the
wares on display, like an airport face-scanning checkpoint, could trace their
lineage to his work.
A physicist, Dr. Atick is one of the pioneer entrepreneurs of modern face
recognition. Having helped advance the fundamental face-matching technology in
the 1990s, he went into business and promoted the systems to government agencies
looking to identify criminals or prevent identity fraud. “We saved lives,” he
said during the conference in mid-March. “We have solved crimes.”
Thanks in part to his boosterism, the global business of biometrics — using
people’s unique physiological characteristics, like their fingerprint ridges and
facial features, to learn or confirm their identity — is booming. It generated
an estimated $7.2 billion in 2012, according to reports by Frost & Sullivan.
Making his rounds at the trade show, Dr. Atick, a short, trim man with an
indeterminate Mediterranean accent, warmly greeted industry representatives at
their exhibition booths. Once he was safely out of earshot, however, he worried
aloud about what he was seeing. What were those companies’ policies for
retaining and reusing consumers’ facial data? Could they identify individuals
without their explicit consent? Were they running face-matching queries for
government agencies on the side?
Now an industry consultant, Dr. Atick finds himself in a delicate position.
While promoting and profiting from an industry that he helped foster, he also
feels compelled to caution against its unfettered proliferation. He isn’t so
much concerned about government agencies that use face recognition openly for
specific purposes — for example, the many state motor vehicle departments that
scan drivers’ faces as a way to prevent license duplications and fraud. Rather,
what troubles him is the potential exploitation of face recognition to identify
ordinary and unwitting citizens as they go about their lives in public. Online,
we are all tracked. But to Dr. Atick, the street remains a haven, and he frets
that he may have abetted a technology that could upend the social order.
Face-matching today could enable mass surveillance, “basically robbing everyone
of their anonymity,” he says, and inhibit people’s normal behavior outside their
homes. Pointing to the intelligence documents made public by Edward J. Snowden,
he adds that once companies amass consumers’ facial data, government agencies
might obtain access to it, too.
To many in the biometrics industry, Dr. Atick’s warning seems Cassandra-like.
Face recognition to them is no different from a car, a neutral technology whose
advantages far outweigh the risks. The conveniences of biometrics seem
self-evident: Your unique code automatically accompanies you everywhere. They
envision a world where, instead of having to rely on losable ID cards or on a
jumble of easily forgettable — not to mention hackable — passwords, you could
unlock your smartphone or gain entry to banks, apartment complexes, parking
garages and health clubs just by showing your face.
Dr. Atick sees convenience in these kinds of uses as well. But he provides a
cautionary counterexample to make his case. Just a few months back, he heard
about NameTag, an app that, according to its news release, was available in an
early form to people trying out Google Glass. Users had only to glance at a
stranger and NameTag would instantly return a match complete with that
stranger’s name, occupation and public Facebook profile information. “We are
basically allowing our fellow citizens to surveil us,” Dr. Atick told me on the
trade-show floor.
(His sentiments were shared by Senator Al Franken, Democrat of Minnesota and
chairman of the Senate subcommittee on privacy, technology and the law.
Concerned that NameTag might facilitate stalking, Mr. Franken requested that its
public introduction be delayed; in late April, the app’s developer said he would
comply with the request. Google has said that it will not approve facial
recognition apps on Google Glass.)
Dr. Atick is just as bothered by what could be brewing quietly in larger
companies. Over the past few years, several tech giants have acquired
face-recognition start-up businesses. In 2011, Google bought Pittsburgh Pattern
Recognition, a computer vision business developed by researchers at Carnegie
Mellon University. In 2012, Facebook bought Face.com, an Israeli start-up.
Google and Facebook both declined to comment for this article about their plans
for the technology.
Dr. Atick says the technology he helped cultivate requires some special
safeguards. Unlike fingerprinting or other biometric techniques, face
recognition can be used at a distance, without people’s awareness; it could then
link their faces and identities to the many pictures they have put online. But
in the United States, no specific federal law governs face recognition. A
division of the Commerce Department is organizing a meeting of industry
representatives and consumer advocates on Tuesday to start hammering out a
voluntary code of conduct for the technology’s commercial use.
Dr. Atick has been working behind the scenes to influence the outcome. He is
part of a tradition of scientists who have come to feel responsible for what
their work has wrought. “I think that the industry has to own up,” he asserts.
“If we do not step up to the plate and accept responsibility, there could be
unexpected apps and consequences.”
‘Not an Innocent Machine’
A few uses of face recognition are already commonplace. It’s what allows
Facebook and Google Plus to automatically suggest name tags for members or their
friends in photographs.
And more applications could be in the works. Google has applied for a patent on
a method to identify faces in videos and on one to allow people to log on to
devices by winking or making other facial expressions. Facebook researchers
recently reported how the company had developed a powerful pattern-recognition
system, called DeepFace, which had achieved near-human accuracy in identifying
people’s faces.
But real-time, automated face recognition is a relatively recent phenomenon and,
at least for now, a niche technology. In the early 1990s, several academic
researchers, including Dr. Atick, hit upon the idea of programming computers to
identify a face’s most distinguishing features; the software then used those
local points to recognize that face when it reappeared in other images.
To work, the technology needs a large data set, called an image gallery,
containing the photographs or video stills of faces already identified by name.
Software automatically converts the topography of each face in the gallery into
a unique mathematical code, called a faceprint. Once people are faceprinted,
they may be identified in existing or subsequent photographs or as they walk in
front of a video camera.
The technology is already in use in law enforcement and casinos. In New York,
Pennsylvania and California, police departments with face-recognition systems
can input the image of a robbery suspect taken from a surveillance video in a
bank, for instance, and compare the suspect’s faceprint against their image
gallery of convicted criminals, looking for a match. And some casinos faceprint
visitors, seeking to identify repeat big-spending customers for special
treatment. In Japan, a few grocery stores use face-matching to classify some
shoppers as shoplifters or even “complainers” and blacklist them.
Whether society embraces face recognition on a larger scale will ultimately
depend on how legislators, companies and consumers resolve the argument about
its singularity. Is faceprinting as innocuous as photography, an activity that
people may freely perform? Or is a faceprint a unique indicator, like a
fingerprint or a DNA sequence, that should require a person’s active consent
before it can be collected, matched, shared or sold?
Dr. Atick is firmly in the second camp.
His upbringing influenced both his interest in identity authentication and his
awareness of the power conferred on those who control it. He was born in
Jerusalem in 1964 to Christian parents of Greek and French descent. Conflict
based on ethnic and religious identity was the backdrop of his childhood. He was
an outsider, neither Jewish nor Muslim, and remembers often having to show an
identity booklet listing his name, address and religion.
“As a 5- or 6-year old boy, seeing identity as a foundation for trust, I think
it marked me,” Dr. Atick says. To this day, he doesn’t feel comfortable leaving
his New York apartment without his driver’s license or passport.
After a childhood accident damaged his eyesight, he became interested in the
mechanics of human vision. Eventually, he dropped out of high school to write a
physics textbook. His family moved to Miami, and he decided to skip college. It
did not prove a setback; at 17, he was accepted to a doctoral program in physics
at Stanford.
Still interested in how the brain processes visual information, he started a
computational neuroscience lab at Rockefeller University in Manhattan, where he
and two colleagues began programming computers to recognize faces. To test the
accuracy of their algorithms, they acquired the most powerful computer they
could find, a Silicon Graphics desktop, for their lab and mounted a video camera
on it. They added a speech synthesizer so the device could read certain phrases
aloud.
As Dr. Atick tells it, he concluded that the system worked after he walked into
the lab one day and the computer called out his name, along with those of
colleagues in the room. “We were just milling about and you heard this metallic
voice saying: ‘I see Joseph. I see Norman. I see Paul,’ ” Dr. Atick recounts.
Until then, most face recognition had involved analyzing static images, he says,
not identifying a face amid a group of live people. “We had made a
breakthrough.”
The researchers left academia to start their own face-recognition company,
called Visionics, in 1994. Dr. Atick says he hadn’t initially considered the
ramifications of their product, named FaceIt. But when intelligence agencies
began making inquiries, he says, it “started dawning on me that this was not an
innocent machine.”
He helped start an international biometrics trade group, and it came up with
guidelines like requiring notices in places where face recognition was in use.
But even in a nascent industry composed of a few companies, he had little
control.
In 2001, his worst-case scenario materialized. A competitor supplied the Tampa
police with a face-recognition system; officers covertly deployed it on fans
attending Super Bowl XXXV. The police scanned tens of thousands of fans without
their awareness, identifying a handful of petty criminals, but no one was
detained.
Journalists coined it the “Snooper Bowl.” Public outrage and congressional
criticism ensued, raising issues about the potential intrusiveness and
fallibility of face recognition that have yet to be resolved.
Continue reading the main story
Dr. Atick says he thought this fiasco had doomed the industry: “I had to explain
to the media this was not responsible use.”
Then, a few months later, came the Sept. 11 terrorist attacks. Dr. Atick
immediately went to Washington to promote biometrics as a new method of
counterterrorism. He testified before congressional committees and made the
rounds on nightly news programs where he argued that terrorism might be
prevented if airports, motor vehicle departments, law enforcement and
immigration agencies used face recognition to authenticate people’s identities.
“Terror is not faceless,” he said in one segment on ABC’s “World News Tonight.”
“Terror has measurable identity, has a face that can be detected through
technology that’s available today.”
It was an optimistic spin, given that the technology at that early stage did not
work well in uncontrolled environments.
Still, Dr. Atick prospered. He merged his original business with other
biometrics enterprises, eventually forming a company called L-1 Identity
Solutions. In 2011, Safran, a military contractor in France, bought the bulk of
that company for about $1.5 billion, including debt.
Dr. Atick had waited 17 years for a cash payout from his endeavors; his take
amounted to tens of millions of dollars.
In fact, some experts view his contribution to the advancement of face
recognition as not so much in research but in recognizing its business potential
and capitalizing on it.
“He actually was one of the early commercializers of face-recognition
algorithms,” says P. Jonathon Phillips, an electronics engineer at the National
Institute of Standards and Technology, which evaluates the accuracy of
commercial face-recognition engines.
Ovals, Squares and Matches
At Knickerbocker Village, a 1,600-unit red-brick apartment complex in Lower
Manhattan where Julius and Ethel Rosenberg once lived, the entryways click open
as residents walk toward the doors. It is one of the first properties in New
York City to install a biometrics system that uses both face and motion
recognition, and it is a showcase for FST Biometrics, the Israeli security firm
that designed the program.
“This development will make obsolete keys, cards and codes — because your
identity is the key,” says Aharon Zeevi Farkash, the chief executive of FST.
“Your face, your behavior, your biometrics are the key.”
On a recent visit to New York, Mr. Farkash offered to demonstrate how it worked.
We met at the Knickerbocker security office on the ground floor. There, he posed
before a webcam, enabling the system to faceprint and enroll him. To test it, he
walked outside into the courtyard and approached one of the apartment complex
entrances. He pulled open an outer glass door, heading directly toward a camera
embedded in the wall near an inner door.
Back in the security office, a monitor broadcast video of the process.
First, a yellow oval encircled Mr. Farkash’s face in the video, indicating that
the system had detected a human head. Then a green square materialized around
his head. The system had found a match. A message popped up on the screen:
“Recognized, Farkash Aharon. Confidence: 99.7 percent.”
On his third approach, the system pegged him even sooner — while he was opening
the outer door.
Mr. Farkash says he believes that systems like these, which are designed to
identify people in motion, will soon make obsolete the cumbersome,
time-consuming security process at most airports.
“The market needs convenient security,” he told me; the company’s system is now
being tested at one airport.
Continue reading the main story
Mr. Farkash served in the Israeli army for nearly 40 years, eventually as chief
of military intelligence. Now a major general in the army reserves, he says he
became interested in biometrics because of two global trends: the growth of
densely populated megacities and the attraction that dense populations hold for
terrorists.
In essence, he started FST Biometrics because he wanted to improve urban
security. Although the company has residential, corporate and government
clients, Mr. Farkash’s larger motive is to convince average citizens that face
identification is in their best interest. He hopes that people will agree to
have their faces recognized while banking, attending school, having medical
treatments and so on.
If all the “the good guys” were to volunteer to be faceprinted, he theorizes,
“the bad guys” would stand out as obvious outliers. Mass public surveillance,
Mr. Farkash argues, should make us all safer.
Safer or not, it could have chilling consequences for human behavior.
A private high school in Los Angeles also has an FST system. The school uses the
technology to recognize students when they arrive — a security measure intended
to keep out unwanted interlopers. But it also serves to keep the students in
line.
“If a girl will come to school at 8:05, the door will not open and she will be
registered as late,” Mr. Farkash explained. “So you can use the system not only
for security but for education, for better discipline.”
Faceprints and Civil Liberties
In February, Dr. Atick was invited to speak at a public meeting on face
recognition convened by the National Telecommunications and Information
Administration. It was part of an agency effort to corral industry executives
and consumer advocates into devising a code for the technology’s commercial use.
But some tech industry representatives in attendance were reluctant to describe
their plans or make public commitments to limit face recognition. Dr. Atick, who
was serving on a panel, seemed to take their silence as an affront to his sense
of industry accountability.
“Where is Google? Where is Facebook?” he loudly asked the audience at one point.
“Here,” one voice in the auditorium volunteered. That was about the only public
contribution from the two companies that day.
The agency meetings on face recognition are continuing. In a statement, Matt
Kallman, a Google spokesman, said the company was “participating in discussions
to advance our view that the industry should make sure technology is in line
with people’s expectations.”
A Facebook spokeswoman, Jodi Seth, said in a statement that the company was
participating in the process. “Multi-stakeholder dialogues like this are
critical to promoting people’s privacy,” she said, “but until a code of conduct
exists, we can’t say whether we will sign it.”
The fundamental concern about faceprinting is the possibility that it would be
used to covertly identify a live person by name — and then serve as the link
that would connect them, without their awareness or permission, to intimate
details available online, like their home addresses, dating preferences,
employment histories and religious beliefs. It’s not a hypothetical risk. In
2011, researchers at Carnegie Mellon reported in a study that they had used a
face-recognition app to identify some students on campus by name, linking them
to their public Facebook profiles and, in some cases, to their Social Security
numbers.
As with many emerging technologies, the arguments tend to coalesce around two
predictable poles: those who think the technology needs rules and regulation to
prevent violations of civil liberties and those who fear that regulation would
stifle innovation. But face recognition stands out among such technologies:
While people can disable smartphone geolocation and other tracking techniques,
they can’t turn off their faces.
“Facial recognition involves the intersection of multiple research disciplines
that have serious consequences for privacy, consumer protection and human
rights,” wrote Jeffrey Chester, executive director of the nonprofit Center for
Digital Democracy, in a recent blog post.
“Guidelines at this stage could stymie progress in a very promising market, and
could kill investment,” Paul Schuepp, the chief executive of Animetrics, a
company that supplies mobile face-recognition systems to the military, recently
wrote on the company’s blog.
Dr. Atick takes a middle view.
To maintain the status quo around public anonymity, he says, companies should
take a number of steps: They should post public notices where they use face
recognition; seek permission from a consumer before collecting a faceprint with
a unique, repeatable identifier like a name or code number; and use faceprints
only for the specific purpose for which they have received permission. Those
steps, he says, would inhibit sites, stores, apps and appliances from covertly
linking a person in the real world with their multiple online personas.
“Some people believe that I am maybe inhibiting the industry from growing. I
disagree,” Dr. Atick told me. “ I am helping industry make difficult choices,
but the right choices.”
A version of this article appears in print
on May 18, 2014,
on
page BU1 of the New York edition with the headline:
