At the end of the Digital Production Buzz interview, I touched briefly upon an analogy comparing 'black-box' video encoders with so-called 'quantitative' stock trading systems used by the large investment banks to manage capital.

Statistics is a form of soft-science which attempts to mathematically describe the phenomenal world, usually by averaging a large number of numerical descriptions/samples to determine trends/patterns which are used, in combination with heuristic algorithms to make decisions about individual events, such as whether to buy/sell or hold a stock.

Statistics, however useful, tell you absolutely nothing about a single sample, or a group of samples. The reason that these heuristically based trading systems are profitable is because the large investment banks can afford to cover short term losses enroute to a larger gain from a predicted trend. It is only because they are embracing a large statistical pool of data that the systems can be profitable.

Thus, an automated trading system is a form of statistical process control, where feedback from the system is filtered statistically, and the results used to control the trading.

This is directly analagous to black-box video compression tools. These tools gather statistical data from one or more passes through the video, and employ this data by running heuristic algorithms to attempt to control the bitrate, predominantly by controlling the amount of compression, or Q (in MPEG parlance), applied to a particular scene, GOP or frame.

Once again, by understanding the nature of statistics, we can anticipate that a statistical black box compression tool, over a wide range of material and many encodes, generally speaking is going to produce good results. However, this does not guarantee that you are going to get good results for a particular encode, scene or group of frames.

Just as a human trader, equipped with both statistical tools, past experience and knowledge of the market can make better-informed decisions about how/when to trade, so then, can we expect that a human compressionist, equipped again with statistical tools, past experience and first-hand visual knowledge of the material (ie. 'knows' particular piece of footage which needs to be compressed) will be able to out-perform a purely automated or statistical system. To return to the analogy of the large investment bank's trading system, what makes that work is huge amounts of capital to risk, deep pockets, and the ability to ride out unwanted losses to reap a later reward.

This rationale is what gets us up in the morning at Digigami, and is the main reason that our encoding tools can produce quality and bitrates competitive with other 'modern' codecs (as mentioned on the show).

Long before I did RoboHelp, which came well before video compression in my life, I worked in the field of industrial process control, specifically for natural gas pipelines (college summer job), and then later for heavy manufacturing where they melt sand into glass and spin it like cotton candy into sheets of home building insulation.

There is no question that statistical methods can be used to augment human insight and experience. We look at it as a way to provide the compressionist/operator with a view of the process which they otherwise could not see. When you cannot see something clearly, you cannot make reasoned, rational decisions about it. Instead, the process becomes mired in myth, legend and religious views.