# Exploratory data analysis

Exploratory data analysis (EDA) is about looking at data to form hypotheses worth testing, complementing the tools of conventional statistics for testing hypotheses[1]. It was so named by John Tukey.

## EDA development

Tukey held that too much emphasis in statistics was placed on statistical hypothesis testing (confirmatory data analysis) more emphasis needed to be placed on using data to suggest hypotheses to test. In particular, he held that confusion of the two types of analysis and employing them on the same set of data can lead to systematic bias owing to the issues endemic in testing hypotheses suggested by the data.

The objectives of EDA are to:

Tukey's books were notoriously opaque, and so several attempts were made to popularise his EDA ideas. Prominent among these was the Statistics in Society (MDST242) course of The Open University.

Many EDA techniques have been adopted into data mining and are being taught to young students as a way to introduce them to statistical thinking.

## Techniques

There are a number of tools that are useful for EDA, but EDA is defined more by the attitude taken than the techniques used.[2]

The principal graphical techniques used in EDA are:

The principal quantitative techniques are:

Graphical and quantitative techniques are:

## History

Many EDA ideas can be traced back to earlier authors, for example:

The Open University course Statistics in Society (MDST 242), took the above ideas, and merged them with Gottfried Noether's work, which introduced statistical inference via coin-tossing and the median test.

For details of the above, see John Bibby's book HOTS: History of Teaching Statistics.

## Software

• CMU-DAP (Carnegie-Mellon University Data Analysis Package, FORTRAN source for EDA tools with English-style command syntax, 1977)
• Fathom (for high-school and intro college courses)
• LiveGraph (free real-time data series plotter)
• TinkerPlots (for upper elementary and middle school students)

## Bibliography

• Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1985). Exploring Data Tables, Trends and Shapes. ISBN 0-471-09776-4.
• Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1983). Understanding Robust and Exploratory Data Analysis. ISBN 0-471-09777-2.
• Tukey, John Wilder (1977). Exploratory Data Analysis. Addison-Wesley. ISBN 0-201-07616-0.
• Velleman, P F & Hoaglin, D C (1981) Applications, Basics and Computing of Exploratory Data Analysis ISBN 0-87150-409-X

## References

1. "And roughly the only mechanism for suggesting questions is exploratory. And once they’re suggested, the only appropriate question would be how strongly supported are they and particularly how strongly supported are they by new data. And that’s confirmatory.", A conversation with John W. Tukey and Elizabeth Tukey, Luisa T. Fernholz and Stephan Morgenthaler, Statistical Science Volume 15, Number 1 (2000), 79-94.
2. "Exploratory data analysis is an attitude, a flexibility, and a reliance on display, NOT a bundle of techniques, and should be so taught.", John W. Tukey, We Need Both Exploratory and Confirmatory, The American Statistician, Vol. 34, No. 1 (Feb., 1980), pp. 23-25.
• Leinhardt, G., Leinhardt, S., Exploratory Data Analysis: New Tools for the Analysis of Empirical Data, Review of Research in Education, Vol. 8, 1980 (1980), pp. 85-157.