An intelligence agency is a necessary but very dangerous institution for a democracy, because it relies on secrecy and borders on illegality. For it to remain under control, it has to be shaken up periodically, which hasn't happened to the CIA in a long time. The very fact that they feel entitled to drive a president from office (what they're doing right now and probably with success) indicates that its time to replace the CIA with something else. That goes for the Dems as well, because there's no way it can be guaranteed that the agency won't do such a thing to one of their presidents too.

CIA's institutional structure produces dreadfully watered down, groupthink "analysis." Individual analysts are fine -- I know some of them, and they're really smart people. But bring on the process of preparing a report, and individual insight turns into group cliches.

The occasional B team experiments proved the value of competitive intelligence. Steve Cambone's DoD shop is proving valuable to DoD, even if it is having little effect on CIA (other than perhaps prompting some senior officials to leak info on the Wilson/Plame affair).

You're absolutely right -- we need to think more carefully about how we analyze intelligence. Competitive analysis by the various branches of the government couldn't hurt.

There are 5 steps in the acquisition and use of intelligence. Step 5 -- the political leaders make a decision -- has next to no correlation with the goodness of the first 4 steps.

You can improve the product all you want, but it you don't improve the consumer, it's waste effort.

It strikes me as silly whenever people who literally do not know what they are talking about presume to discuss important matters such as the gathering and analysis of intelligence regarding the activities of foreign nations that want to keep their secrets secret. All sorts of folks -- some of them not loons -- claim to know something about what goes on in the CIA (or the NSA, or any of who knows how many other outfits, some of which may not even be known to exist by those outside the inner circle of government), but we here have no way of distinguishing the blowhards from the real experts, do we?

The concept of an "open source" database probably strikes the professionals as a contradiction in terms. How, for example, do you publish information that would, if disclosed, reveal exactly how you got it, from whom, and so on? The cardinal rule AFAIK is, "Protect your sources." Violate that, and you have no sources.

As for "shaking up" the CIA, how do we know that it has or has not happened? All we know is what we are told, or allowed to see. Can we be sure the folks who are telling us the "inside story" are not just nutcases? No. Appearances are deceiving, and if they aren't -- you don't have a good intelligence agency.

The best we can hope for is that over time, we may get a general notion of how well the spooks and their colleagues are doing. We have pretty good reason to believe now that the CIA was misled into thinking the USSR was a stable nation, able to carry on its rivalry with the West. But has it ever occurred to anyone that this "error" might be itself a ruse? Perhaps the CIA was saying all along, "Star Wars is breaking the back of the USSR, keep it up!" We can only speculate.

If we could do more than speculate, if we really had any information that would allow us to evaluate the performance of the CIA and related outfits, our spies and intelligence analysts would not be doing their jobs. So what we do is base our opinions on our own political ideologies, some of which are myths, and sit around the campfire telling tales to each other.

One of the easiest tales to tell is how the spooks are easily deceived, and their bosses are idiots. The CIA nailed Che Guevara in Bolivia, and that looks pretty darn good to me. So I'm willing to tolerate some errors and miscues, given the fact that these folks are groping after data that are being closely guarded by some very nasty people.

Are we really ready to tell the spooks to stand down, open their archives, reveal their sources, expose their analyses to the scrutiny of wannabe spies who don't have the background to make sense of it all?

IMHO, this "shake up the CIA" and "make their work open source" is just a lot of posturing by people who TALK a good game, hint that they know some of the insiders (Ooooooh, I'm IMPRESSED), have some experience that they want us to believe qualifies them to blow hard, or have a Walter Mitty complex.

Back to reality, everybody. We can't all be James Bond, or even an assistant professor of international relations at Podunk State Secretarial College. Come off it, you self-defined experts: you have an ego problem. You want to instruct everyone in a subject you know next to nothing about. You simply aren't qualified to oversee, let alone reform, the CIA.

No, the fact that the USA is a democratic republic does not make you an expert about anything, and it sure doesn't give you a license to sort out an outfit like the CIA. You have even less business lecturing the spy agencies than you do telling the medical schools what subjects to teach, what medications to prescribe, and how to diagnose disease. Your role is that of citizen and voter, which means you get to have your say regarding the choice of the folks who run the country, not sharpshoot the experts they depend on for information. If you don't like the way things are going, find yourself a political candidate whose intellect you respect, and vote for him.

Now go back to your beer and stop pretending to be "connected," or "just come in from the cold," or some other romantic figure from a lousy novel.

Fritz:

Yes, that's what we're ready for. Daniel Patrick Moynihan laid out the case for shutting down the CIA and doing away with governmental secrecy very well. Secrecy has been a complete failure.

Fritz,

it may be possible to use a Google type ranking algorithm to rank the "quality" of open source analysts. A similar phenomenon occurs in the open-source software development world, where reputation is used to rank code patches.

I for one think this idea has merit. Using OpenSSL and PKI certificates, it should be possible to set up a system wherein comments are positively linked to analysts, and the system ranks (possibly using a Bayesian algorithm that initially assumes uniformly bad and/or uninformed commentary, and adjusts upward over time the rankings of those who provide quality analysis) how close analysis comes to subsequent events.

I also see no reason why this couldn't be done with a futures market model with anonymous analysts.

I agree with most of what Fritz said. This is truly a really, really bad idea. Here's what would be exposed:

CNWDI: Critical Nuclear Weapons Design Information. Do you really want to sell your secrets to the highest bidder?

WNINTEL: This used to be a marking used to notify that intelligence assets or methods would be compromised if the information were leaked. This marking has been changed, but the need for protecting sources is still there.

COMSEC: This has to do with communications security, obviously. Now, if you have an open source kind of exchange going on the need for this goes away. Still, how do you protect, say, communications on the battlefield if your techniques and keys are all out in the open.

But it's possible I've misunderstood. I'm certainly open to how you think having it all out in the open is a good thing.

As for "shaking up" the CIA, how do we know that it has or has not happened?

In the past I worked on various intel and defense systems for over 15 years -- besides a Top Secret DOD clearance, I was cleared to 4 SCI areas.

Fritz is actually the one who's full of crap. I can't really explain why.

Some notes, however.

1) A study indicated that 85% of intel needs could be satisfied from open sources. Obviously, this varies by area.

2) Given that the government's procurement process, with it's emphasis on tight control, resembles the management processes of the Soviet Union Politburo and Mao's Great Leap Forward, you might suspect that it is not ..er..agile. I recall a project years ago that was seven years long (two years over schedule). When it was finally deployed , the customer was extremely unimpressed. The user interface was per specifications developed early in the program. In the meantime, Apple's MacIntosh and Microsoft's Windows had been deployed. It hurts when a system costing several hundred million dollars looks worst than what a soldier knows he can buy at CompUSA.

3) Custom made, one of a kind superprojects done in the style of the Pyramids are not competitive with the rapid innovation of Dell, Microsoft, Silicon Valley,etc. The government can't spread R&D costs over a hundred thousand customers.

4) I recall another example in which several organizations spent months discussing how to integrate their databases dispersed over the world. At one conference, The program lead had just finished presenting the upteenth
revision to the requirement spec and promised an initial prototype in about a year. Someone then
followed with a demonstration of the new Netscape Internet browser. There was an embarrassed silence at the end as the participants realized that Netscape had just put out a product --immediately available for roughly $50 -- that accomplished much of what they wanted and were planning to spend almost a $million to develop with a delivery date several years into the future.

5) The Intel budget is classified. However, during a period of openness in the Clinton administration, Aviation Weekly put out a supposedly rough draft based on partial leaks.
What the AW budget indicated was that several huge "technical systems" --for imaging,etc. -- were receiving tens of $billions while the budget for the CIA Clandestine Service (Humint) was less than a $Billion.

A little thought will show why -- defense and intel contractors like Boeing, Raytheon, Lockheed Martin,etc receive tens of $Billions every year to build the huge technical systems. They therefore spends $millions/year lobbying Congress to keep the gravy train running. By contrast, the Humint spies can hardly lobby Congress and they don't donate shit. Guess whose programs get the majority of the loot?

Unfortunately, we did not have enough human ears deployed on the ground in Central Asia --spreading enough bribes -- to get much warning of Sept 11. And the much vaunted "national technical systems" never ..er.. saw the terrorists coming.

6) Regarding Humint, I have one word: Aldrich Ames This article by a former Clandestine Service officer describes the goat fucks in the Humint world: http://www.theatlantic.com/issues/2001/07/gerecht.htm

Dated a few months before Sept 11, the above report predicted that Osama Bin Ladin would run rings around the CIA.

By contrast, Richard Shelby, the Republican Chairman of the Senate Select Committee on Intelligence , told this to the Washington Post this a few months prior to Sept 11:

"But Shelby, vice chairman of the Senate Select Committee on Intelligence, said in a recent interview that bin Laden is the one who's on edge.

"He's on the run, and I think he will continue to be on the run, because we are not going to let up," Shelby said.

"I don't think you could say he's got us hunkered down. I believe he's more hunkered down," Shelby said. "He's moved and tried to be one step ahead of our intelligence on where he might be. He knows he's hunted, and he's not exactly strolling down the streets of London or Paris or Berlin, shopping."
Ref: http://nucnews.net/nucnews/2001nn/0106nn/010626nn.htm#350

It's always nice to know that we have competent and intelligent politicans running the Intelligence Community.

7) Some of the more hilarious goat fucks are described in the annual reports put out by the
Congressional Intelligence Committees --see,e.g,
http://thomas.loc.gov/cp108/cp108query.html
(At the top you select the particular Congress --select 106 -- and then select Intelligence from the list of House Committees.

A few years ago, the House Committee(HPSCI) discovered that huge sums had been spent to collect certain data but nothing had been done to disseminate the data to the users.

HPSCI rewarded this incompetence in it's customary manner --it threw more money at the problem-- never wondering whether giving people huge sums to fix trainwrecks might be a reason why the trainwrecks keep occurring.

Of course, HPSCI members don't really have the time for such introspection --since they have to spend a lot of time calling Intelligence Contractors in order to solicit campaign donations.

8) HPSCI did a particularly hilarious report on NIMA (National Imaging and Mapping Agency) a few years ago -- see http://www.fas.org/irp/agency/nima/commission/toc.htm .
After describing in detail numerous NIMA trainwrecks, the report tactfully explained the basic problem:
"NIMA does not have the organic capability or the experienced technical leadership to successfully acquire TPED, nor can it "get there from here," in time, using normal government practice. There is no help on the horizon because neither the NRO nor NSA has the talent to spare. If the US is to have a good chance of achieving a TPED capability to give the nation the information edge in the 21st century, special steps must be taken to ensure success.

The Commission recommends creation of an Extraordinary Program Office (EPO) armed with special authorities of the Director of Central Intelligence and the Secretary of Defense, augmented by Congress, and staffed beyond ceiling and above "cap" through an heroic partnership between industry, NIMA, and the NRO. "

In other words, NIMA didn't have the intellectual capital to do it's job.

9) Given that the American people pays huge sums every year to support the Intelligence Community --and given that they are the ones who suffer the consequences of huge intel failures like Sept 11 -- I think it is in the national interest for them to discuss the subject.

10) PS David Perrod's slight hysteria re
exposure of "Critical Nuclear Weapons Design Information" is hilarious given that the
Nuclear Weapons FAQ is out there. See
toward bottom of http://nuketesting.enviroweb.org/hew/ .

The design of primitive nuclear weapons was leaked several years ago when the Russian archives on the Rosenberg Case were published and disclosed the intel reports passed to Russia from spies at Los Alamos.

Nukes based on U235 are simple shoot-one-subcritical-mass-at-another designs --Little Boy wasn't even tested before being dropped. The problem is that it takes a huge,expensive process to separate small amounts of the U235 isotope out from the chemically identical U238 --and U238 makes up almost 99% of natural uranium. Look at the huge Oak Ridge complex.

Nukes based on Plutonium , by contrast , must use the more complex implosion design --but this design was developed in months under primative conditions at Los Alamos in 1940 when it was discovered that the gun-design wouldn't work with plutonium ( Gun design wouldn't work because the chain reaction would start when the two subcritical masses of plutonium were still inches apart and heat from this early start would blow the two masses apart before the reaction could progress very far--a fizzle.) The Nuclear Weapons FAQ explains how this problem was solved. Plutonium can be generated in a U238 pile and easily separated from U238 --although it is a bitch to work with.

First off, let me tell you where I sit, before I tell you where I stand -

I'm not an active-duty intel officer. I am not affiliated with the CIA, NSA, DIA, NRO, or any other intel body. I am, however, a grad student, working toward a Master's degree in Strategic Intelligence, and am currently at the mid-point in the process of gaining a commission as a Naval Reserve Intel Officer. In short, I won't claim to speak authoritatively on practical intel matters, but know enough to speak to the issues raised here to a certain extent.

There have been several good points made here from those on both side of the issue. Open source intel is, quite frankly, better than most alternatives in many cases. Look at the issue of the Iraqi nuclear program; there is ample open source documentation regarding the technical specs of both Iraq's EMIS efforts, and their gas centrifuge programs. These documents are extremely detailed, and provide the clearest look that we've gotten at Hussein's original plans for uranium enrichment (to which he shifted after the Israelis disposed of his breeder program back in '81).

That having been said, there will ALWAYS exist a need for secrecy in the world of intelligence. It's part and parcel of what makes intelligence a valuable commodity. If your enemy has access to the same info you do, you have lost any advantage that the information in question would have provided.

The more important question here, as has been stated before, is that of analysis. As Kevin mentioned before, the structure and culture of the CIA tends to lead to a "watering down" of analysis, such that, even if the info at hand points strongly to one potential event over another, there is pressure exerted by the the culture of the CIA, and by the State Department to dress the info in such a way as to render it as innocuous as is possible. There is little or no incentive to stick one's neck out in the realm of analysis - rather, there is a great emphasis on uniformity, even at the expense of accuracy, to some extent.

One of the age-old problems with intel analysis is its traditional reliance on strictly intuitive means, rather than truly analytical methods. There are many in the community working to change this (one of the best is Dr. Jonathan Lockwood, whose Lockwood Analytical Method of Prediction is rapidly gaining acceptance within the world of intel analysis. Change will not come quickly or easily, but is crucial to the improvement of our intelligence community as a whole.

Jared:

Presumably your enemy knows what you know, though not necessarily that you know it. Consider just one example: in Michael Beschloss's book on the U2 incident, he writes that Ike understood the USSR to be years or even decades behind us militarily, but figured that if we didn't confront the world with that knowledge the Soviets wouldn't try to catch up. That was a disaster. The Soviet system, always rickety, would have imploded thirty years earlier if they'd tried to pull even.

At some point, if we have faith in liberal democracy, why not just lay out in full detail what a disaster the other systems being tried elsewhere are and all the measures they're having to take to cover up the fact?

If you want to talk about Open Source Intelligence (OSINT), you have to start by reading Robert David Steele at http://oss.net. Steele's a former CIA station chief & head of MI8 (Marine Intelligence) now in private practice, & he's been the primary proponent of OSINT since the mid 90's. The website's not been updated recently, but it's still got everything you need to know on the subject, & then some.

s/n:r

You still need the spies to verify intelligence, it seems to me. What does it mean to get a tip over the internet? The most anyone could say is that a given tip is likely or unlikely. It also seems susceptible to tampering, much as Poindexter's market may have been, although I thought that idea was worth a try.

From the admittedly little I've read, the CIA wasn't even using it's existing tools in the Middle East. One thing I read said the number of Arab-speaking, or Arab-American agents was pitiful, and that recruiting practices were not addressing the matter.

Just doesn't seem to be time to throw the baby out.

Chris:

But they've gotten too much wrong to believe thay're capable of verifying it.

Chris K
A big problem, from what has appeared over the past few months is that your Intel crowd don't understand the mindset of the present enemy. How they chose people to conduct interrogations is amazing.
To have an agent who refused to record his "interviews" with people of his religious persuasion, continue working!
To be so naive as to believe that adherents of a "Mindset" so rigid would keep an oath they made to gain citizenship of a State they "hate".
Nowadays while there might still be things to be watched in Russia, the new challenge needs a deep understanding of what is a totally alien culture.

So, Don, your contention is that we don't have to protect any of our nuclear weapons design information because the design is essentially unchanged over the last 60 years or so? I really can't have this discussion with you, if that's what you think. If it were true, the unauthorized transfer of weapons technology to the Chinese that was only about 40 years old wouldn't have made any sense.

And if you detected hysteria, I can't have this discussion with you, either; you really need to have this discussion with your psychoanalyst. Something about projection, I believe.

I know it probably came as a shock to those who read your post that intel hardware costs far outstrip humint costs. I was utterly unflabbergasted by that, though. Spy satellites are frigging expensive. It's hard to get the kind of overhead views they give you from humint, though.

And, finally, your assertion that 85% of intel needs could be satisfied by open source is something I don't care to contest one way or the other. However, if true, it means that 15% of our needs can't be. It seems a little cavalier to hand-wave roughly a sixth of the total intel, though. What did you have in mind for that?

Second sentence above should have read "If it were true, the hysteria over the unauthorized...".

In other words, why would we be upset over the Chinese getting a design that's essentially out in the open to start with?

To respond to Perron's strawman, I never suggested that some intelligence should not be kept secret -- some information can only be obtained by expensive covert or clandestine operations. The effectiveness of those operations is lost if their existence is revealed --e.g, by exposure of their product.

Nonetheless, here is what I understood to be the gist of Fritz's post --that
outsiders cannot discuss intelligence intelligently , that such discussion cannot be helpful for the intelligence community, that the intelligence community is largely competent and technically advanced, that outsiders cannot contribute useful ideas, that
intelligence is beyond the ken of mere mortals, that the intelligence community is well managed , that most classified info is really classified and unknown to the world. I think that these points are questionable. (choke ).

As I recall, a former CIA analyst testified at Robert Gates' confirmation hearings that Gates had
helped to greatly mislead Congress over the Soviet Union's capabilities because such deception advanced Reagan's military agenda and budget requests.

Somehow, it never occurred to anyone to look at open source info and ask why the Reagan was spending 7.5% of the US GDP on defense while Germany and Japan were only spending 3% and 1% respectively --even though Germany and Japan were mere miles from the "evil empire" vice being on the other side of two oceans.

Given that Reagan-Bush dumped a massive federal debt on taxpayers equal to 4 times each taxpayer's annual payment, Fritz might understand why ordinary citizens might want to ask if that expense was necessary. Any such discussion drags in intelligence.

Similarly,Fritz might recall that the Director and Deputy Director of the NRO were relieved of command by CIA Director Deutsch after it was discovered that they had secreted monies in a slush fund (I believe the amount was eventually $4 Billion) and that roughly $300 million of that money had gone to building a lavish headquarters building in Fairfax County, outside Washington. And that the Oversight Committees only found out about this when Fairfax residents started asking who owned such a huge , beautiful building?
(PS The relieved Director was immediately hired by a large intelligence Contractor. Who would have received the funds in the secret slush fund --e.g., via cost overruns on classified programs -- is left as an exercise for the reader.)

Re nukes, of course there is still some information that is and should be classified --the detailed design of small powerful thermonuclear warheads that can fit on cruise missiles, for example. But my understanding is that DOE was still classifying the early design of nukes long after the information had been exposed by the Russian. DOEs overclassification keeps important information from reaching those Homeland Security types worried about how terrorists might develop a primitive nuke.

I think the community suffers a huge overhead cost and burden from overclassification of information. That cost/burden is not apparent to its managers but it exists.

Overclassification also has malign effects on national security in ways other than what's described above --often in ways not obvious. Extreme slowness in innovation due to difficult, burdensome, limited, slow communications mechanisms can be one.

Another instance is Important information not reaching those with a strong need to know in those cases where the government bureaucracy changes the information dissemination/classification mechanisms far more slowly than changes occurring in the world and within the organization of the national security community. How many first responders --firemen and cops -- got WMD informations in the weeks after Sept 11, for example?

As another example, the US Critical Infrastructure is very vulnerable because of poor computer security -- as we have seen in Microsoft announcements every week.

Computer security, in a lot of ways, requires strong encryption. However, NSA was trying to withhold encryption technology from the commercial world over the last 30 years as our computerized infrastructure was developed.

The commercial companies threw in the towel and it's now very expensive to correct the national security problem that a too-narrowly focused NSA helped create.

Fritz might remember that the Intelligence Community is ultimately a civil service bureaucracy similar to the Postal Service --but possibly without the Postal Service's glamour and sense of mission.

Don:

Thanks for the clarification. By the length and force of your original post on the topic of weapons design, it appeared to me that you thought all weapons design classification unnecessary. Now I know otherwise.

I don't have any argument with you over some information being unnecessarily classified. I've seen data items classified secret, yet all the information needed to calculate those items not classified at all. And it certainly doesn't surprise me that the government may have mishandled large sums of money, given that what the money was supposed to have been spent on was classified, or some such.

It's that 15% I'm worried about.

Just a thought fellows for a partial solution. Take a look at the simputer, strip the cell phone stuff out and put a Wifi link in (IEEE 802.11b). Add a solar recharge system. Stick hotspots all around Iraq, Afghanistan, wherever US boots are occupying ground and in need of an informant network. Give these things out and put important information on them like job info, where to get rations, curfew information, etc. Give them email and internet access. And finally, make it simple to send messages to intel to report on the bad guys.

You've just created a system where you don't have to leave your house to help the good guys, you have to have the damn thing otherwise you're going to be unemployed and hungry, you have access to FAQs in the native language so you can learn about democracy, plans to transition to a local government, and all the other stuff that we'd like to teach them.

Finally, you've created a system which will serve intelligence purposes in two ways. 1st, honest people who are scared can send messages when they wouldn't dare come to a camp gate. Second, it's inevitable that the bad guys will try to take advantage of the fact that this communication system exists and will attempt to use it themselves. Poor deluded folks, their traffic's going to be monitored and their hardware addresses and physical locations are going to be known in a way that simply doesn't happen on the regular Internet.

Given another decade of Moore's law, this is likely to get practical even for use by illiterates who would have to use voice command.

Mr. Perron:

Isn't the question less whether people know how to build them than whether we let them?

Do both.

Keep the whole secret apparatus, spies, etc. Fritz and his allies are right -- you can't compromise secret sources. But ...

You could set up an open-source free-for-all, anyway.

You use them as a check on each other. The open source thing helps you decide where to focus your secret assets. The secret sources allow you to what areas you want the open source operation to develop, where to focus -- and of course you drop the odd red herring in the open source to keep the bad guys guessing.

I don't see why it has to be an either/or. And it does seem to me that a way to cheaply gather masses of observations and insights from non-official sources would be worth having.

The point here seems to be whether a new type of capability should be added. It seems to me a good case has been made that it should be.

This is a fine discussion. For information:

9,800 pages, all past Proceedings for all Open Source Intelligence (OSINT) conferences, have been loaded to Archives at www.oss.net.

I am no expert on intelligence by far, but I am current military and I work as a security officer for our intelligence community. It seems to me that what many of you with insight into the intel community complain about is our failure to utilize the information and technology that is available through the commercial sector. I would agree, although there are definitely some guidelines we need to stick to. We can't just go putting the newest and coolest stuff on our networks... any flaws in the software could put our protected networks in danger. What I would propose is that we start holding our agencies accountable to continue to increase our intel gathering and analysis capabilities. I believe we need to push our intel personnel (especially leadership) to "bring something new to the table" every now and then. For so long I have seen crotchety old intel guys who still rely on antiquated procedures and equipment to collect and disseminate intel. I'm not advocating throwing away all their valuable experience, but force them to evolve... faster, better & cheaper as the saying goes. If terrorists can figure out how to keep their plots secret, while still communicating worldwide on unsecure networks, perhaps there is something we can learn from them.