Like almost all government agencies, DARPA answers to the executive branch and receives funding from congress.
There are many parts of the US government that are heavily influenced by term-level time scales for example NASA comes up with a 10 year plan every 8 years. Several DARPA programs operated from at least 2012 or 2013 to 2020, which means that they successfully crossed administrative boundaries. However, term scale incentives may have begun to break down - Starting in 2001, DARPA directors began to sync up with presidential administrations.
Because DARPA Program managers pull control and risk away from both researchers and directors, the DoD itself doesn’t have direct control over which programs DARPA is running. However, the DoD does put pressure on the DARPA director to work on areas that are currently relevant to the military, like counter-terrorism and insurgent warfare in the 2000’s or jungle warfare in the early 1970’s.
An opinionated director is the lynchpin keeping DARPA focused on long-term things that nobody is asking for (yet) (Einstellung Effect.) At several points in its history, the director went to bat to keep DARPA from being dissolved or absorbed into the military as a more “normal” R&D organization. This secret meta-dependence on the director makes the increasing politicization of the director potentially detrimental to DARPA and there is evidence that DARPA has shifted away from long-term disruptive work in conjunction with the probable director politicization.
Originally congress funded ARPA through a lump sum but over time has shifted to requiring a budget for each program, including what they plan to accomplish for the year. You know this is bullshit because the plans have statements like
Advance development of design tools for the optimization of collaborative problem solving performance in human-machine systems and systems-of-systems. In 2003, congress investigated and effectively defunded the Total Information Awareness Program.
Abstractly, opacity seems important because if your source of money is constantly looking over your shoulder and judging what you’re doing, you’re going to take actions that look good. A source of money like congress that has many opinions exacerbates the incentive to ‘play it safe’ because making sure any action you take is acceptable to a committee is almost as bad as the committee determining those actions. Einstellung Effect. Coincident with congress digging more into program specifics, more and more DARPA programs are becoming classified. It is complete speculation but it hints at a tension between DARPA’s desire to keep things opaque and congress’ desire to have oversight thanks to a trust breakdown possibly tied to increasing politicization.
Over time, DARPA has become less opaque and the director has become more coupled to the current administration. If you Pay attention to DARPA’s informal process and ignore formal process it leads to the conclusion that Opacity is important to DARPA’s outlier success and DARPAs director is important. Of course, both of these can be abused but You can’t cut off just one tail of a distribution.