We live in a time of major innovation, and with some extremely powerful, long-term views hinting at an interesting future — Elon Musk’s next-phase plans, Jeff Bezos’ 20-year-old mega-success, and Google’s push for data-driven autonomous transportation, to name just a few.

Yet, new products are fickle beasts. Developing and launching them is not for the faint of heart. It takes a special kind of crazy to venture out into uncharted territories, nevermind while whistling and with a bounce in your step. The chances for success are always uncertain. What can seem robust at conception may emerge with unforeseen problems, inducing a long stretch of sleepless nights. What might seem unimpressive on the drawing board may blaze an amazing trail once it interacts with the market’s oxygen.

After what I consider to be a major pivot period away from a lot of feature experimentation within journals and books (2008-2012), the scholarly communications market seems to be in the throes of a new round of innovators creating products and businesses outside of journals and books — from infrastructure offerings to new journal publishers to new measurement and collaboration tools. Our innovation period is more nascent, but it is growing in sophistication, from Digital Science to Crossref to a number of other companies — big and small — delivering new ideas to the market.

A common thread in this new round of innovation is that it is coming from outside traditional organizations. There are a few exceptions (bioRxiv, NEJM Knowledge+), but these seem the exceptions that prove the rule. And this may be due to the difficulty of new products breaking free from the gravity of larger parent organizations.

Within academic and scholarly publishing, we tend to have cultures that are mostly about incrementalism, so it’s no surprise that product development can struggle in these environments. In larger companies, there is often a sense of being “on guard” within product innovation teams, with a lingering worry that the larger machine may suddenly change its trajectory and support level.

Within a larger organization, product development can appear to be sexy and exciting compared to daily operations. This can attract a mix of dispositions to the product development exercise, some of which aren’t suited to the activity. It’s as if all those who have learned to drive now want to get behind the wheel of a long-haul rig without any additional training. Or if all pilots insisted they could be astronauts, thinking space flight is just flight, only higher.

Because innovation teams tend to work better when they’re small and highly coordinated, temperament matters. There are those who are built for the vagaries of product creation, and those who are not. Making something from nothing is work of a special kind. It’s where creativity meets optimism meets panic meets pragmatism meets persistence. And product development comes in different sizes, from bottle rockets to full-on Apollo programs.

To prevent being hamstrung, teams involved in the creative process usually need some protection, granted by an executive or provided structurally, or both. Even then, incrementalists can insinuate themselves into the process and compromise it, usually unwittingly and with the best intentions.

Product development within a larger organization has asymmetrical politics to it, which can undermine efforts, as well. Operational teams have funding, authority, a clear pecking order, and institutional validity, while the product development team may have, at best, two of the four. Over the long arc of developing a truly daring and innovative product, this asymmetry can prove detrimental to morale and corrosive overall. A sleepless young developer suddenly has a child enter his life, and takes a safer, slower job in the larger organization in order to gain the stability he and his new family now crave. A firebrand product manager gets worn down by her operational counterparts’ elitism as they talk about their larger teams and meetings with senior executives. The product development executive gets promoted out of the role, leaving the team to fend for itself. The rebel alliance is always at risk when it exists inside the empire.

Established organizations also have established business models, and if they’re willing to spend money on product development, these established business models are probably working pretty well. This creates an opportunity and a threat for innovation initiatives. The opportunity is the funding. The threat is an inherent impatience for revenues — after all, if the current business is able to generate $xxx,xxx dollars with just a tweak in pricing, why can’t the new innovation perform the same magic? Why is it taking so long? Where is the payoff? Annual budget cycles only add to these pressures.

Some larger organizations solve this innovation dilemma with acquisitions — monitoring the innovation flourishing outside, and then acquiring the new thing when it’s proven itself. Elsevier is the most obvious example of this, with Mendeley and SSRN, but other examples exist, and the pace of acquisitions seems likely to increase. Growth remains a necessity for any business, and with more constraints on organic growth, organizations will be looking to acquire the growth curves that exist outside their current boundaries.

One of the biggest trends in product development this century is the technology offering. Often, these offerings are positioned as “open source,” with some hinting that this includes the business model and not just the technology maintenance model. In our space, new offerings from Coko, PLOS (with their Aperta submission system), and eLife (their Continuum journals platform) may challenge the assumption that product development leads to appreciable new revenues. More on this later, as it gets to the distinction between a product and a business, and between technology and finance.

Some companies solve the innovation puzzle via spin-offs of their own, such as the Canadian Medical Association’s new Joule initiative, the American Institute of Physics’ separation of publishing into AIPP, or the Biochemical Society’s Portland Press Publishing.

After years of leading product development within larger organizations, I’m now leading a startup that is a spinoff, itself. Being in a startup brings product development front and center — because developing the products we sell now and those we plan to sell in the future is all we have.

This singular focus has many benefits, with a lack of competing priorities being chief among them. A sense of excitement and urgency is also a benefit — after all, the entire enterprise hinges on the success of new products, so headway feels great and challenges have to be addressed and not deferred.

Within this high-paced environment of everything needing some level of attention, comfort in the face of ambiguity comes to the fore. My organization is currently launching a number of different products. Some have traditional business models — annual subscriptions, for instance — while others are free, and still others to come will have mixed models we’ll be trying for the first time. We’re going to have to learn quickly what works and what doesn’t, and be willing to change our original approach if needed. Or do we hold our ground and wait? Or do we find a way to do a little of both? Everything is provisional. The old maxim of “and is better than or” comes into play quite frequently.

These kinds of uncertainties aren’t something everyone likes to deal with. But some people find it exhilarating. It’s like solving a puzzle, a mystery, a financial exercise, and an engineering problem all at once, all while aligning a great team to find the solutions. Ultimately, it will be these people who solve the problems and find the solutions.

For example, we’re currently launching a free infrastructure service to help libraries and publishers collaborate on access details — things like IP address updates, Shibboleth, link resolvers, and branding. It’s built on the premise of a professional network, wherein connections drive authority and trust. It took a long time to develop, and we’re happy with it. But there’s always uncertainty.

You might think a free product like this would be embraced by the market quickly, but I’m not convinced it’s that simple. It may take longer than we think for “the network effect” to take hold. There are other challenges — security layers that are necessary and elegantly deployed may still create barriers to adoption; the ability to describe how a network motif solves professional communication issues; and the elusive shift of habits from a traditional work paradigm to a new workflow paradigm.

For a free product, obviously price isn’t the barrier. In fact, inertia and uncertainty might be the biggest barriers. Inertia creates a switching cost all its own — that is, why deviate from my routine to embrace your offering? The status quo is a powerful force in the market. Uncertainty about the benefits can also slow adoption. Overcoming these are a major hurdle for any new offering, and a true test of value. There is also the pace at which networks scale — it’s akin to the first fax machines, which only became really valuable when then were a lot of fax machines. How will this network gain scale?

Open source products tend to work when they are part of a revenue-generating business — that is, value-added elements are built on top of the open source offering, whether services, support, enhancements, or premium features. The Mozilla Foundation runs its browser as an open source software offering, but has multiple revenue streams, including a multi-million-dollar contract (once with Google, now with Yahoo/Verizon) for placement as its default search engine. At one point, this contract was worth a whopping $300 million. So, despite their open source technology positioning, such products inevitably transition into businesses, becoming “freemium” offerings economically, while remaining “open source” as pertains to the code base. The market positioning here is more and more common as buying habits change.

We also return to the question of a product vs. a business, which is something the investors on Shark Tank often use to justify an investment decision. They generally don’t want to fund a product, preferring to fund a viable business with multiple products. Unfortunately, many entrepreneurs enter the Tank with only a product idea, and not a business vision. Products don’t bring scale and risk diffusion with them natively, while businesses can and should. It’s worth reviewing Joe Esposito’s post distinguishing a business, a product, and a feature. These distinctions are important along many dimensions, and eliding the differences is only asking for trouble down the road.

Finally, we arrive in the zone where I think a lot of product development teams find themselves captured by some mysterious gravity  — they underestimate the important breakthrough power of marketing and sales. Most new products require more marketing for a longer time and with more branches of activity than most product teams appreciate. The blind spot here can be large, especially if a product development team mostly consists of people who build product, rather than people who sell product. In the former case, they may be satisfied when the product is built, not when the product is sold. It raises an existential question: Do you really have a product if you haven’t sold it?

This issue is especially important within larger organizations, which don’t necessarily easily assign major marketing resources to new initiatives. Yet, only by engaging the market can the value of the new offering be understood, refined, and ultimately used to enhance the business’ revenues and future. It becomes a Catch-22 — there is not enough money to spend on marketing to make sales to generate revenues, so insufficient money is spent on marketing, few sales are made, and the Catch-22 is complete.

Product development should be aimed to get beyond the point of sale, which is where it can slip the bonds of inertia and gravity. This may be why new products from new companies seem more likely to succeed — there is less competition for marketing dollars, and there is no choice but to adapt, move forward, and generate revenues. It may take 3-5 efforts to fine-tune the approach, but there is a resilience to necessity. In larger organizations, a small setback can be amplified into defeat by the culture and politics and competing priorities of that organization. After all, there is always the core business to embellish.

Maybe Yoda knew the secret to product development success — “Try not. Sell . . . or sell not. There is no try.”