Big data and the future of farming

Mark Henry
Australian Farm Institute

Of late, the term ‘big data’ has generated a lot of buzz and a lot of questions in the agriculture sector. By some accounts, the implementation of big data applications will effectively enable farmers to be replaced by autonomous machines in the not too distant future. Alternatively, others are claiming that big data is simply a grab for more information by big corporations, and that it will not result in any productivity gains or benefits for farmers. Exactly what is being referred to when people use the term big data, and what potential implications this might have for Australian agriculture are questions which the following paper attempts to address.

Big data refers to the aggregation of large sets of data that enables the detection of trends and patterns.

Perhaps the most readily available example of the use of big data is found in supermarket loyalty card schemes. Major Australian retailers utilise demographic information about individual consumers such as their location, age, gender and family status, in combination with the detailed purchase information that is generated each time they use their loyalty card, to build up very detailed statistical information about the behaviour of their customers. This is then used to assist a broad range of management decisions ranging from the choice of products placed together on retail shelves to the tailored promotional material that is sent to particular customers in specific locations.

Big data applications have emerged in agriculture as a result of the ever-growing volume of digital information that is now being generated by modern farm machinery, and through the development of electronic livestock tracing systems, remote sensing, electronic weather data and even the use of smartphone mapping and other applications. The ever-decreasing cost of generating, storing and processing information makes it feasible to collect this digital information into large databases, and to analyse it to identify patterns and trends that can be used to assist future decision-making.

Strictly speaking, the use of the term big data refers to computational systems that are utilised to analyse very large collections of data – think billions and trillions of pieces of digital information. A relevant example might be a hypothetical project to collect all the yield maps generated by all the grain harvesters operating in Australia in any one season, to combine that information with all the weather and soil data relevant to the growing season in each paddock, and to use that to come up with a formula to predict grain yields from soil and weather data. This would obviously require very large data storage and digital processing capacity, well beyond the capability of an individual farm business.

At the farm level, the use of the term big data is probably not appropriate. What is often being referred to is actually ‘digitally-enhanced agriculture’, which is the generation, collection and use of digital information on-farm to control machinery, record production information and to assist in farmer decision-making. The generation of this type of information at an individual farm level is a potential starting point for the development of big data applications relevant to the management of farms, but at this stage there are only very limited instances where true big data applications are utilised in Australian farm decision-making.

This growth of digitally-enhanced agriculture does bring with it the potential for a seismic increase in the role of data science in driving agricultural productivity growth and informing agricultural management decisions, in order to lift farm productivity. In fact, a recent survey of farmers in the United States (US) conducted by the Hale Group (2014) in conjunction with the University of Iowa revealed that yield improvements and reduced costs were the highest motivators of digital technologies adoption by farmers (Figure 1).


Figure 1: Motivators to try prescription agriculture.

Source:  The Hale Group (2014).

The productivity enhancements of most interest to farmers typically include yield increases in crops, genetic gains in livestock, and greater efficiency of input use, leading to cost savings.

In a report to the Iowa AgState Group released in December last year, the Hale Group (2014) provided estimates of the potential gains that are available to farmers from the application of best-practice digital agricultural technologies and the insights arising from big data. In calculating the gains available, the researchers calculated the differential between the performance of ‘technology pioneers’, and those yet to adopt the technology. The report found it is feasible to achieve a yield gain of five to 10 bushels of corn per acre, plus savings on input costs.

A yield gain of five to 10 bushels of corn per acre (with a current corn price of US$3.50/bushel) amounted to the potential of US$18 to US$35 per acre increase in gross proceeds. The researchers also noted nitrogen fertiliser savings amounting to between US$25 and US$30/acre per acre, resulting in total benefits of between US$43 and US$65 per acre. These were offset slightly by costs of between US$3 and US$10 per acre, which was the cost of the digital agricultural storage and analytical services.

Whether these gains are available under the low-input cropping systems utilised on Australian broadacre cropping farms is as yet uncertain. A typical US corn crop utilises much higher rates of fertiliser and chemical inputs than is the case in Australia, is grown under less variable weather and soil conditions, and the US Government also provides high-precision weather and soil data that can be utilised in digital farming systems. Australian agriculture, with its diversity of products and production environments, does not always lend itself to a ‘copy-paste’ adoption of US agricultural systems. Nevertheless, the potential of significant productivity gains does appear to be available.

The productivity potential that is luring farmers to adopt digital technologies is also spurring a symbiotic development of decision-support tools by farm input suppliers and software developers. Optimising operations through digital agriculture necessarily alters the decision-making process on-farm. This creates the potential for more remote management, or to franchise a highly skilled manager across a much greater number of acres. It also creates the potential that farmers will become more reliant on ‘black box’ decision-making processes dictated by embedded algorithms.

Can big data replace farm managers?

Determining when and what operations need to be executed has long been the exclusive role of a farm manager, but software tools that have been enabled through the application of big data are now moving into this space. In order for this to occur, large aggregated data sets are needed to unlock the power of data analytics. These include historical data sets as well as up-to-date weather and soil data, seed variety testing and benchmarked harvest data. In the US in particular, the routine collection of this data by farmers, machines and government agencies over the past decade has created the critical mass of information which can now be utilised for this purpose.

Suppliers and analytics companies have begun to build data interoperability into their business models. In the US, proprietary data storage and analytical companies are setting themselves up as central hubs, and encouraging software providers to compete in supplying data tools and services. From the perspective of software companies, this allows access to a pan-industry data repository from which data analytics can extract operational insights and ultimately begin to determine when and what operations need to be executed. Companies already well established in this space include:

  • SST Software which enables a farmer or their advisor to map, store and combine soil test data, weather information and past field performance in order to develop management zones within a field and apply different management treatments to those using variable rate application (VRA) technologies and precision guidance systems.
  • The Climate Corporation’s Nitrogen Advisor application provides decision-support information about the consequences of farmers’ proposed soil nitrogen applications at the field level in order to assist corn farmers to optimise nitrogen fertiliser use.
  • Fruition Sciences, a Californian company, has developed big data techniques to assist decision-making in vineyards. Sensors record and transmit sap flow in vines every 15 minutes to a computer application which combines that information with weather data and calculated rates of evapotranspiration and vine stress to develop an optimal irrigation regime.

While the input reductions achieved have been impressive and attractive to farmers, the use of these tools represent a progressive shift away from management based on the subjective assessment of experienced mangers toward algorithmic based management decisions. In some respects, it could be considered to be a first step along a path leading to the wholesale industrialisation of agriculture in the same way that manufacturing industries were industrialised with the development of the production line and mass production by the former farmer Henry Ford over 100 years ago.

The Holy Grail for software companies at present is the objective analysis of data to produce probabilistic or deterministic crop advice for farmers and their advisors. There are only a small number of software companies that have developed, or are developing this capability, but it is not currently incorporated in the majority of existing farm software systems. Farm profitability is still very much contingent on the knowledge and experience of the human manager, but change is underway.

Employing big data as a farm manger

Good farm management is a valuable commodity. It’s often said that the difference between a good farmer and the rest of the pack can come down to a few days. There is a lot profit in knowing when to plant, spray, irrigate and harvest. Add to this the responsibility of selecting seed varieties and animal genetics, the ability to consistently monitor operations and identify problems quickly and a picture emerges of how much farm revenue is tied to quality managerial skills. Research and benchmarking studies identify that ongoing farm profitability is highly contingent on the skill of the manager. Top tier mangers can reliably avoid losses in bad years and outperform in good years. Lack of quality mangers, or rather the outlay needed to identify, employ and incentivise the top tier farm managers has been blamed for poor returns in corporate farming both here and overseas. Technology that can largely automate farm management could have major ramifications for agriculture in the future. Even partial automation of management skills has the potential to disrupt farming, given the possibility it creates to spread the skills of a single manager across many more hectares. There is a parallel with the introduction of the combine harvester, which did not completely replace the labour required to harvest grain, but did enable a single skilled operator to harvest much greater areas and volumes of grain than was previously possible.

Automation of farm managerial decision-making creates the possibility of a future where a small number of ‘super’ managers will operate in conjunction with specialised analysts and software providers to make farm management decisions on vast tracts of land. These will need to be supported by a labour force consisting of relatively low-skilled farm labourers, and perhaps specialist contractors. This model is not that far-removed from the current model that operates in some Australian cropping regions, where unskilled seasonal labour is imported for seeding and harvesting.

The potential changes driven by the application of big data to agriculture are not confined to the farm. Major agrichemical and bioscience companies report that their employment focus has shifted from plant and animal scientists to data analysts and software engineers, many of whom have had absolutely no experience in the agriculture sector.

Does big data demand big agriculture?

If big data applications significantly reduce the requirement for skilled on-farm management, corporate farm owners may find it much easier to expand their farming portfolios. A centralised team of analysts, software providers and regional managers could conceivably leverage big data technology to monitor and orchestrate farming operations over multiple properties. As planning shifts off-farm, skill requirements for on-farm labour will be reduced. The spectre of job polarisation looms large under such structures.

It could be argued that these development have already occurred in the case of farming systems for which intensive data collection is feasible, and where there is a high degree of control that can be exercised over production factors. Intensive piggery and poultry operations, and glasshouse horticultural production are relevant examples, and these sectors have certainly evolved from consisting of large numbers of family scale businesses to ones which are dominated by large-scale corporate enterprises.

For now, however, the family farm appears likely to be the predominant model of farming into the future. True big data farming systems for broadacre farm businesses are only in their infancy, and are often targeted towards assisting farmers to make specific critical decisions, rather than the full range of day-to-day decisions that are made by farm owners and managers. Computer applications are still designed for use by farmers and their advisors, and software providers emphasise that ease of use and simplicity are key elements of the success on any available software tool.

A major limitation is the availability of large volumes of robust and comprehensive data detailing all of the wide ranging factors that are known to contribute to the success of a particular farming venture. Until a large volume of such data becomes available, the ability of software engineers to develop robust and reliable decision-support tools is limited.

Compared to the rest of the economy, agriculture has a disproportionate number of decision-makers. As of 2014 there were 2.73 workers per agricultural business. For the entire economy there were 5.15 workers per business. Management skill remains a significant overhead expense for the agriculture industry, whereas other economic sectors have found ways to reduce these costs.

Farmers tend to be romanticised somewhat, and not without reason. They operate their business in a highly volatile environment, often in relative physical isolation, and without access to services and infrastructure considered absolutely essential by most other members of the community. That said, it would be a mistake to assume that the digital disruption that is currently affecting people in all walks of life will not extend to the farm sector. It would also be a mistake to consider that community perceptions of farming will insulate the sector from some of the more disruptive effects that are likely to arise from the development of these technologies. After all, domestic consumers have already demonstrated that their sentimental perspectives of farming did not inhibit their enthusiasm to save 20 cents per litre of milk, despite the obvious havoc it wreaked amongst dairy farmers and processing companies.

Despite the cautions, there is of course a tremendous upside likely to emerge from the application of big data to agriculture, with the potential of a new leap forward in farm productivity as management transitions from the flock and paddock level to the individual animal and square metre level. It is, however, worth being mindful of the fact that as digital agriculture develops and emerges, there will be winners and losers.


Hale Group, The (2014), The digital transformation of row crop agriculture, AgState electronic survey findings, December, available at:

Image:  Valley Irrigation 

Back to November 2015 index