2 Tips When Asked To Estimate Your Data Projects Duration

3 minute read

“ … and how long will that take?”

This is ALWAYS hard for a technical expert to answer. It makes one feel twitchy.

By their nature data projects are innovative, unpredictable and covering new ground.

You are being asked to create something new. Something that has never been done this way before.

The requirements appear solid but experience tells you that you will not get the full picture until you really get into it.

Then the client or your boss asks you how long the work will take and you know they are going to hold you to whatever comes out your mouth next…

In fairness, it is a valid question (for them)

Having solid time commitments enables people to schedule activities around your work.

They enable project owners to set wider organisational expectations by letting them know when to expect outcomes from their investment.

They ensure potentially never ending projects, complete.

Unfortunately, accurately estimating the duration of data projects is notoriously hard – get it wrong and you will be letting people down.

In the wider world of software development, they use theories around short “Sprints” to calibrate a team’s expected work rate.

The idea is to refine the volume of work targeted in each 1 or 2-week sprint until they get a predictable sprint cadence.

That is great for them but personally I’m not so sure data projects fit the “agile” approach.

We are building one-off prototypes most of the time.

Data projects are a puzzle with lots of promising answers competing against each other that you slowly refine through logic and experimentation.

There are too few similar enough repeated activities too easily predict the pace.

If things go well you can actually get great results very quickly.

Other times you can be ridiculously delayed (often through no fault of your own) by something that one could never have predicted – typically to do with some silly kind of data wrangling or data access issue.

Data projects require a lot of thinking. A lot of ideas. And a lot of testing of concepts.

Frankly it is a bit messy. Especially at the start.

You therefore need to have enough time and space to do this properly.

Equally, you really need to be adding value back to the business within a meaningful time frame otherwise they are going to lose faith.

What you want are solid time estimates that you can confidently give to the team.

Time estimates that are not too pessimistic to be useless, yet not too optimistic that any slight hiccup results in you having to pull all-nighters.

How I deal with this is to try to desensitise and de-risk the time estimates whilst still giving the team a meaningful answer.

Try these two simple things:

1- Try Chunking Your Work Into 1 Month’s Worth of Effort

When we work on complex projects we try to split them down into smaller pieces.

This is works perfectly for when you are trying to build things. Often the more granular the better.

It works less well for your project planning because when things go wrong, your carefully considered hourly timing plan blows up in your face.

Make this easy.

Breakdown your activities down but not by as much as if you were about to actually sit and develop something.

Consider instead what you could do in about a week.

For example, I would give myself about a week to:

  • develop a concept dashboard design including trialling various visualisation types
  • develop a data capture form including the error checking, I/O and element designs
  • design, model and test an initial data structure
  • understand, test and experiment with a third party API
  • conduct a feedback session including pre-event activities such as walkthrough videos, slides, meeting admin and post-event activities such as feedback analysis

Think chunky.

Think achievable.

Then combine these elements together into about a months worth of scope and estimate on the total.

Once you get the hang of it you can start to do this very quickly.

You will gain on some activities and lose on others but, I find, over a month, everything tends to even itself out.

2- Only Ever Quote A Range

A range estimate gives you a larger landing area for successful delivery.

It also reduces the likelihood of you letting people down. 

I use two-week ranges.

For a month’s projects, I would therefore estimate 3 to 5 weeks.

I have found this creates the right balance of offering meaningful value to the business whilst not tying you to set unrealistically short time scales.

If you find yourself hitting things in 3 weeks (can’t say this has ever happened much but I’m sure you’re better than me!) then you can start adding more in next time.

Clarifications: I find 1 month is a minimum, this works equally well for 4-6 weeks or 5-7 weeks etc.

In summary, use the chunky estimates, minimum of a month with two-week ranges.

The bonus tip

If there is ever a stand-off between scope and time, always reduce scope to fit the time.

Whilst people will be disappointed on the surface it is much better to have this conversation at the beginning rather than at the end.

I found you can manage peoples expectations nicely by “officially” deferring some scope to “version 2” but promising to include what you can from that scope if you have time at the end or before the next meeting on longer projects. 

Most good clients are happy if you set these expectations at the start. It also gives you an opportunity to shine, if you choose.

How do you approach estimating data project timelines? Do you have another method that works well for you? Drop me a note at (hello@yourdatadriven.com) and let me know how you approach this.


Whenever you're ready, there are 4 ways I can help you:

  1. Get 1-1 Help From Samir – Sometimes you just need an expert race engineer to get you back on track. Get your questions answered, the best approaches made clear and your goals achieved, with my support.
  2. Master Your Tyres course [link] - Go beyond tyre pressures and guesswork. Uncover the science, tools and techniques to optimise your tyre setup, and your driving style, so you can confidently get more grip, better handling and faster lap times, on any track, in any conditions.
  3. Motorsports Data Analysis course [link] - Ever wondered what insights you might be missing? Turn the data you're already collecting into accurate, personalised guidance to help you drive faster.
  4. Get your business in front of the best motorsports eyeballs in the world by sponsoring Ahead of the Curve [link]