Data Lifecycle in Agriculture


You were introduced to the data lifecycle concept in  🧑‍💻⁠⁠Module 1: Data 101⁠ . In this section, we will dive deeper into how the data lifecycle applies to your farm operation.
  

The data lifecycle follows the evolution of a data point, or a dataset, as it is collected, stored, used and ultimately deleted. It is a useful framework because it organizes and structures the sequence of operations that all data goes through during its lifespan. That said, it is important to note that your farm as a whole doesn’t go through the discrete steps of the data lifecycle in a neat, sequential fashion. Instead, the data lifecycle applies separately to each dataset, or type of data, you manage on your farm. As a result, you are certain to have different datasets simultaneously at different stages of the data lifecycle.


Making the Link:  Planning

We know that the production cycle plays a central role in the operation of a farm business. For farmers growing crops outdoors, that production cycle is an annual event with one crop per year. Other growers working in greenhouses, or with livestock may have different production cycles based on the nature of their farm.   

As we discussed earlier, planning and preparation is a critical part of data management. As a result, it is important to create a strong link between your data management practices and your farm’s crop or livestock production cycle. As you plan for your next crop, it is an opportunity to think about your approach to data using the data lifecycle. For example, making a good plan for data collection before seeding allows for the right data to be collected throughout the year so you have what you need to analyze your yield relative to the cost of production and growing conditions.  

In the following section we will walk through the steps of the data lifecycle and look at some of the key considerations to take into account when you are planning your data collection and management approach in parallel to your other annual, or seasonal, farm planning activities.  


Phase 1: Acquisition

Data acquisition, or data collection, is a critical step in the data lifecycle because how it is done goes a long way to determining what you can do with your data. As we have discussed at length, high quality data is needed to support good data-driven decisions, and to get a return on the investment you put into your farm data.

Therefore, rather than rushing to get started with data acquisition it is critical to put a plan together:
  • What types of data do you need to collect?
  • How much data is needed and how frequently will you collect the relevant data?
  • What kinds of tools (i.e. devices, equipment, software) are needed to acquire the data?


Data Acquisition Methods

In  🚜⁠⁠Module 2: Data in Agriculture⁠ , we catalogued the many different types of agriculture data. As you can imagine, there are also many ways to capture farm data including in-field sensors, farm machinery, drones, mobile phones, among others. In some cases your farm may also receive data that has been collected by service providers, or your trusted advisors.

The table below provides an overview of different data collection methods and their relevant advantages.
Data Acquisition Method
Advantages
Manual Data Entry
Simple and no special equipment needed.

Useful when network connectivity is not available.  

Data can be logged manually into a physical notebook or a digital tool (eg. phone, tablet, or computer).

Automated
Reduced need for labour for data entry.
Sensor-Based Technologies
Reduced data input errors that can occur from manual data entry. 

Increased volume of data, or even real-time data, can be created (e.g. soil, weather, machinery).

Remote Sensing
(e.g. drone and satellite imagery)

Ability to monitor large geographic areas with minimal, or no, additional labor.

Low cost which allows for repeated measurements over the growing season.



Data Acquisition Best Practices

Clearly define the data that needs to be collected.
What are your goals? What data is necessary to achieve these goals?

Clearly defining the scope and purpose of any data you are collecting will reduce the risk of collecting unnecessary data that can be time-consuming and costly. 

Choose a data collection method that best suits the needs of your farm.
Do you plan to collect data using automated sensors or manual data entry? 
What type of tools will you use to collect data? 

The type of technology you use for data acquisition can significantly influence the volume, quality and the utility of the data gathered.  

Data standards, protocols and commonly used formats.
Are there established protocols for collecting this data?  
Are there standards or commonly used data formats that could be used?

Using standardized data acquisition methods can help ensure data is collected in a consistent manner across different locations, time periods, and datasets. 
It also increases the likelihood that the data can be used by other stakeholders in the agri-food value chain.



Phase 2: Storage

Properly storing your data makes it available for processing and future use. The way your data is stored is also relevant to safeguarding your farm's data against unauthorized access and potential cybersecurity risks (see  🔐⁠⁠Module 6: Cybersecurity⁠ ).  

In many cases, data storage is built into the service provided by agtech companies (i.e. farm management software, weather stations apps, etc.). In this scenario you will access the data managed by that company via their app or website, rather than interacting directly with the data files. 

That said, you may also choose to store your data manually if you want to have a backup copy, or need to use the data outside of the relevant agtech software tool. Manual data storage can be done either on-site or off-site in the cloud:

  • Desktop Computer On-site, or Local Storage: Storing data files on a local device or computer that is physically located on your farm. 

  • Cloud Off-site or Cloud Storage: Cloud storage (see  🌐⁠⁠Data Infrastructure⁠ ) provides added benefits like safeguarding data against local computer failures due to mechanical issues, floods, or fires. Typical cloud storage platforms include  Google Drive ,  Dropbox  and  Microsoft One Drive , among others. 


File Organization

If you are storing and organizing your farm data yourself, a structured file system is important so you can find the data you’re looking for at a later date. The approach you take for organizing your data files doesn’t need to be complicated. It can be as simple as the file tree similar to the example shown in the image below that organizes the files by year and by major data category (e.g. agronomy, equipment, financial).


The Value of Strong Record Keeping

Allan is a farmer in Southern Alberta and typically grows alfalfa hay to sell to local beef producers. He also cultivates lentils and other crops like oats and durum wheat as part of his rotation. After a few years growing alfalfa on a small 100 acre field, Allan recognizes that he isn’t making the income that he needs from this part of his farm. Based on some initial soil testing, he concludes that this is because of the increased salinity of the soil.  In response, he decides to switch to a different crop in that field in the next growing season but checks his historical farm data to confirm this approach by looking at past soil salinity and the relative performance of different crop types under similar conditions.  Good record keeping enabled Allan to make an informed decision in an effort to increase the productivity and profitability of his farm business. 



Phase 3: Processing

At a minimum, a certain amount of basic data processing, or data cleaning, must be done before data can be used. Data cleaning involves detecting, correcting, or removing, corrupt, inaccurate, or irrelevant parts of the raw dataset (see  🗄️⁠⁠Organizing Data⁠  in Module 1) to improve its quality and reliability. 

In the context of agriculture, data cleaning ensures that the data acquired from various sources such as farm machinery, sensors, drones, weather stations, or manual inputs is accurate and consistent.

As a grower, most data processing is likely to be done by the agtech company whose products or services you are using to collect the data, or a trusted advisor. However, it is still important to be aware that data processing is happening, as well as who is responsible for this step of the data lifecycle.



Phase 4: Usage

As we’ve said before, your farm data is only worth something if you put it to use - to inform your farming operations and decision making. This means that usage is the most important phase of the data lifecycle. With that in mind, it is important to start your planning work here so that every other stage of the data lifecycle is aligned with getting the outcome you are looking for from your agriculture data.   

Early in the planning process you can ask yourself a few key questions to get started:
  • What are my goals this season or next year? 
  • What are the major decisions or risks I am facing?
  • How could my farm data be used to help? What specific data do I need to collect? How would I analyze the data to yield usable results? Would it be helpful to collect or acquire additional data?
  • Have I acquired any new equipment, or technology that could be helpful? What type of data does this new technology collect, or generate?  

As you think through these data use questions you will naturally start to engage with questions related to other phases of the data lifecycle:  
  • How am I going to collect that data?
  • How do I ensure that the data I’m collecting for this use is high quality?  How can I check the quality using data coming from another source (e.g. ground truthing)?
  • How is my agtech vendor processing and storing the data?

You may also want to consider using data that was collected in previous years, or historical data. Historical data can help you identify trends and patterns that enhance your decision-making or problem-solving capabilities.  

Here are some examples of how data can be used to support on-farm decision-making:

Using NDVI Imagery Data to Reach Decisions

Normalized Difference Vegetation Index (NDVI) imagery shows the health of plants by measuring how they absorb and reflect different wavelengths of light. NDVI imagery from satellites can be used by farmers and their trusted advisors to identify healthy crops versus those that require more attention.  Historical NDVI imagery can be used for making planting decisions in future growing seasons. You can check out this resource to learn more:  https://gisgeography.com/ndvi-normalized-difference-vegetation-index/ 



Phase 5: Disposal

Data disposal is the final phase of the data lifecycle and refers to the process of safely disposing of data that is no longer needed, or no longer relevant. Disposal is important for data privacy, security, and efficiency, as it costs resources to store data.

Data that is subject to disposal could be data that has reached the end of its retention period, data that is obsolete or duplicated, and data that is subject to legal requirements that require its deletion. Of course, before moving ahead to delete data, it's important to consider whether the data still holds potential value for your farm business.  


Data Disposal Best Practices

Data Type
Examples
Recommended Retention Period
Financial Records
  • Receipts
  • Invoices
  • Tax returns
  • Payroll records
It is recommended to seek guidance of a financial professional (e.g. accountant) to confirm the recommended retention period.
Agronomic Data
  • Soil composition data
  • Crop yields
  • Weather
  • Pest and disease incidence
Additional years of agronomic data are useful to support decision making.

It is recommended to retain agronomic data for 10 years, or as long as is practically possible. 

Legal Documents
  • Land deeds
  • Contracts
  • Leases
It is recommended to seek guidance of a legal professional for the recommended retention period.
Farm Activity Records
  • Planting schedules
  • Harvest yields
  • Livestock health records
In the same way as agronomic data, records of your farming activities are useful to support decision making.

It is recommended to retain this type of data for 10 years, or as long as practically possible.

Equipment Maintenance Records
  • Records of equipment purchases
  • Maintenance schedules
  • Repair logs
The recommended retention period is for the life of the relevant equipment.



Next:  ⁠⁠Summary⁠