Go Back Back to courseware

Data Practices:

1.6 Data Project Post-Mortem

[Use arrow keys to navigate, "s" to show speaker notes, and "f" for fullscreen.]

PDF Print

With Notes

Topics Covered

  1. Questionnaire
  2. Qualitative & Quantitative
  3. Things to Evaluate
  4. Data Process
  5. Example Questionnaires
  6. Action Items

Exercise 0: Answer the
"Big Three"

For a recent data project, or for an existing data workflow, answer the "Big Three" post-mortem questions.

  • What worked that you'd absolutely do again without change?
  • What didn't work that you would never do again?
  • What worked, but you would do again differently? How?

Qualitative & Quantitative

Qualitative is important, but hard to measure

  • The "Big Three" questionnaire
  • "Good" vs "Bad" outcomes
  • "Satisfied" stakeholders

Quantitative is easier to measure, but hard to filter

  • Goals met?
  • Numbers changed?
  • Graphs well

By Your Powers Combined!

  • Combining feedback into a single summary can be hugely beneficial
  • Try to give context with both Quantitative & Qualitative metrics
  • Make sure everyone's voice is included to help spur discussion

Things to Evaluate

Exercise 1: How do you evaluate success?

Based on your current process, what sorts of things do you evaluate to determine success?

What other things should you consider as it relates to data?

Our List

  1. Team Effectiveness
  2. Project Overall Success
  3. Individual Project Goals
  4. Stretch Goals
  5. What worked? What Didn't
  6. Quality of Output
  7. Auditability
  8. Ethical Handling of Data
  9. Data Process

Team Effectiveness

  • Did you have the right people?
  • What methods of communication did you use?
  • Were handoffs and deadlines effectively communicated

Project Overall Success

  • What success metrics did you establish before the project started
  • Did you meet all or most of those metrics
  • Why or why not?
  • What general qualitative sentiments did people have about the project?

Individual Project Goals

  • Break down some of the individual goals of the project
  • How well did you meet these goals?
  • Were there more difficulties in one area than another?

Stretch Goals

  • Were there any stretch goals established for the project?
  • Were you able meet these goals (or could you have)?
  • Were these truly stretch goals?

What Worked? What Didn't?

  • Tools
  • Processes
  • Models
  • Data Sources

Quality of Output

  • How were insights delivered?
  • Who was the target audience?
  • Was it effective?
  • What could have been done to improve?

Auditability

  • Was the project documented end-to-end?
  • Could a new employee follow the process end-to-end without special instruction?
  • Could you follow the exact same steps and achieve the same results (repeatability)?

Ethical Handling of Data

  • What ethical standards do you apply to data work (if any)?
  • What regulations are appropriate for your locality/industry?
  • How well did you meet expectations?
  • Are there ethical considerations that you should apply even if not mandated?

Data Process

What to look for

  1. Kickoff
  2. Source
  3. Profile
  4. Prepare
  5. Explore
  6. Analyze
  7. Report

Examples

data.world

https://goo.gl/SnE8Yk

Structure:

  • Overview
  • Accomplishments
  • Areas for improvement
  • Lessons learned
  • Action items

Shopify

https://goo.gl/G98Rwq

Structure:

  • Project overview
  • Project ratings
  • Project questions
  • Comments

Dexterity

https://goo.gl/q8ismB

Structure:

  • General Info
  • Communication
  • Project management and scheduling
  • Information management
  • Issue management
  • Other comments

Exercise 2: Designing Your Questionnaire

Using the examples, and your own experience, build (or outline) a brief questionnaire that you could use to gather data prior to a post-mortem.

Don't forget action items

"An effective postmortem should generate action items. While it shouldn't be a criminal investigation, they're an affirmative process designed to make us all smarter."

--Ken Norten, Partner, GV

Exercise 3: Socializing You Plan

Now that you have a questionnaire, your original/current approach, and a working knowledge of what is needed, spend 5 minutes sketching out the changes you would like to make. Present your findings to a neighbor.

Want to run a workshop like this at your company?

community@data.world



Don't forget to sign the values and principles! https://datapractices.org/manifesto