During the 1950s, the State Department began to scrutinize public servants in its ranks, methodically scanning personnel files and interviewing suspected threats. The goal was to root out “immoral,” “scandalous” and “dangerous” government employees—people whose personal conduct put the entire nation at risk.

You might think the targets were suspected Communists—after all, it was the height of the Red Scare and Cold War paranoia. But the State Department’s targets weren’t suspected Communists, and the sweep wasn’t run by Joseph McCarthy. Instead, LGBT people were in the crosshairs, accused of unfitness to serve. Condemned as “perverts” and bullied out of their jobs, they were systematically targeted for their sexual orientation.

The period—considered as targeted and as widespread as the concurrent Red Scare—is now known as the Lavender Scare. Between the late 1940s and early 1960s, an unknown number of LGBT employees, likely in the thousands, were driven out of their jobs. Countless others were interrogated and bullied—all in an attempt to purge State of LGBT people.

Historian David K. Johnson gave the period a name in his book, The Lavender Scare: The Cold War Persecution of Gays and Lesbians in the Federal Government. Johnson documents the use of the phrase “lavender lads” to refer to gay men. It was used by tabloids like Confidential and people like Senator Everett Dirksen, who was involved in public hearings related to the Senate purge, and it represented a wider societal tendency to mock and fear LGBT people.

At the time, homosexuality was a crime, and gay people had long hidden their sexualities. After World War II, as cities grew, underground gay cultures began to flourish. Despite the prevailing view of homosexuality as a mental illness and a sign of perversion or criminality, gay people started to find one another at underground bars and clubs. In the meantime, American culture became more sexually conservative even as more and more people became aware of homosexuality. This provoked a backlash, and cities began to more aggressively police sexual expression.

So did the State Department. As the federal government began to persecute suspected Communists, gay people found themselves being targeted. At the time, many people equated Communism with homosexuality—people like Senator Joseph McCarthy, who linked what he considered to be the madness of Communists to the supposed mental imbalances of gay people.

“Many assumptions about Communists mirrored common beliefs about homosexuals,” notes National Archives archivist Judith Adkins. “Both were thought to be morally weak or psychologically disturbed, both were seen as godless, both purportedly undermined the traditional family, both were assumed to recruit, and both were shadowy figures with a secret subculture.”

In an attempt to lock down national security, the State Department began to actively seek out homosexual employees. As Congressional hearings about supposed homosexual activity within the department raged, the intelligence community began interviewing and pressuring for the resignations of suspected gay employees. Investigators looked for supposed signs of homosexuality, like being unmarried, and scrutinized employees’ and potential hires’ voices, mannerisms and dress for stereotypical markers that they might be gay.

“The official rationale wasn’t that homosexuals were Communists, but that they could be used by Communists,” Johnson told The University of Chicago in 2004. “A variation on the blackmail rationale…held that Communists promoted “sex perversion” among American youth as a way to weaken the country and clear the path for a Communist takeover.”

As the search for gay State Department employees intensified, so did the pressure. People were questioned, publicly humiliated and mocked by investigators. They were encouraged to denounce others and report suspected homosexuals. And in 1953, President Eisenhower signed Executive Order 10450, which defined a laundry list of characteristics as security risks, including “sexual perversion.” This was interpreted as a ban on homosexual employees, and even more firings took place. Publicly humiliated and devastated by the loss of their income and their reputations, some even killed themselves.

Others, like Frank Kameny, fought back. Fired in 1957, he petitioned the Supreme Court for relief in recognition of his civil rights. They declined to take the case, so he picketed the White House. He fought to counter workplace discrimination for the rest of his life. Kameny wasn’t the only person galvanized by the public targeting of LGBT people—in 1969, the Stonewall Riots made gay rights a front-page issue, and the movement Kameny helped start and the Lavender Scare helped foment has flourished ever since.

The scare lasted until the 1960s, when investigations slowed. Only in the 1970s was the ban on gay intelligence community members relaxed, and it took until 1995 for another executive order, signed by President Bill Clinton, to explicitly state that the government may not discriminate based on sexual orientation when it comes to granting access to classified information. By then, countless gay people had been reminded for years that their participation in the State Department was not wanted—and that they would be treated as second-class citizens if they tried to serve their country.

Shortly before leaving office, Secretary of State John Kerry made a public apology on behalf of the State Department for the persecution of LGBT employees. “These actions were wrong then, just as they would be wrong today,” he said. The apology has since been removed from the State Department’s website—a reminder that struggles over LGBT rights are anything but relics of the past.

Today, memories of the Lavender Scare are fading as the people it targeted grow older. The experiences of others, who never told their stories for fear of being kicked out of their jobs, too, will never be known.