|
|
| (799 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | ==Data analysis==
| |
| − | Data analysis is the process of inspecting, cleansing, transforming, and modelling data with a goal of discovering useful information, suggesting conclusions, and supporting decision-making.
| |
| | | | |
| − |
| |
| − | '''Data mining''' is a particular data analysis technique that focuses on the modelling and knowledge discovery for predictive rather than purely descriptive purposes, while business intelligence covers data analysis that relies heavily on aggregation, focusing on business information.
| |
| − |
| |
| − |
| |
| − | In statistical applications, data analysis can be divided into:
| |
| − |
| |
| − | *'''Descriptive statistics''',
| |
| − | *'''Exploratory data analysis (EDA)''', and
| |
| − | *'''Confirmatory data analysis (CDA)'''.
| |
| − |
| |
| − |
| |
| − | In particular, data analysis typically includes '''data retrieval''' and '''data cleaning (pre-processing) stages'''.
| |
| − |
| |
| − |
| |
| − | '''Exploratory Data:''' In statistics, the exploratory data analysis (EDA) is an approach to analyzing data sets to summarize their main characteristics, often with visual methods.
| |
| − |
| |
| − |
| |
| − | '''Confirmatory Analysis:'''
| |
| − | In statistics, confirmatory analysis (CA) or confirmatory factor analysis (CFA) is a special form of factor analysis, most commonly used in social research.
| |
| − |
| |
| − |
| |
| − | '''Empirical Research:'''
| |
| − |
| |
| − |
| |
| − | '''Statistical Significance:'''
| |
| − |
| |
| − |
| |
| − |
| |
| − | '''Descriptive Data Analysis:'''
| |
| − |
| |
| − | *Rather than find hidden information in the data, descriptive data analysis looks to summarise the dataset.
| |
| − | *They are commonly implemented measures included in the descriptive data analysis:
| |
| − | **Central tendency (mean, mode, median)
| |
| − | **Variability (standard deviation, min/max)
| |
| − |
| |
| − |
| |
| − | '''Exploratory Data Analysis:'''
| |
| − |
| |
| − | *Generate Summaries and make general statements about the data, and its relationships within the data is the heart of Exploratory Data Analysis.
| |
| − |
| |
| − | *We generally make assumptions on the entire population but mostly just work with small samples. Why are we allowed to do this??? Two important definitions:
| |
| − | **Population: A precise definition of all possible outcomes, measurements or values for which inference will be made about.
| |
| − | **Sample: A portion of the population which is representative of the population (at least ideally).
| |
| − |
| |
| − |
| |
| − |
| |
| − | Types of Variable: https://statistics.laerd.com/statistical-guides/types-of-variable.php
| |
| − |
| |
| − | ===Central tendency===
| |
| − | https://statistics.laerd.com/statistical-guides/measures-central-tendency-mean-mode-median.php
| |
| − |
| |
| − | A central tendency (or measure of central tendency) is a single value that attempts to describe a set of data by identifying the central position within that set of data.
| |
| − |
| |
| − | '''The mean''' (often called the average) is most likely the measure of central tendency that you are most familiar with, but there are others, such as the median and the mode.
| |
| − |
| |
| − | '''The mean, median and mode''' are all valid measures of central tendency, but under different conditions, some measures of central tendency become more appropriate to use than others. In the following sections, we will look at the mean, mode and median, and learn how to calculate them and under what conditions they are most appropriate to be used.
| |
| − |
| |
| − | Mean
| |
| − |
| |
| − | Mean (Arithmetic)
| |
| − |
| |
| − | The mean (or average) is the most popular and well known measure of central tendency.
| |
| − |
| |
| − | The mean is equal to the sum of all the values in the data set divided by the number of values in the data set.
| |
| − |
| |
| − | So, if we have <math>n</math> values in a data set and they have values <math>x_1, x_2, ..., x_n,</math>the sample mean, usually denoted by <math>\bar{x}</math> (pronounced x bar), is:
| |
| − |
| |
| − | <math>\bar{x} = \frac{(x_1 + x_2 +...+ x_n)}{n} = \frac{\sum x}{n}</math>
| |
| − |
| |
| − |
| |
| − | The mean is essentially a model of your data set. It is the value that is most common. You will notice, however, that the mean is not often one of the actual values that you have observed in your data set. However, one of its important properties is that it minimises error in the prediction of any one value in your data set. That is, it is the value that produces the lowest amount of error from all other values in the data set.
| |
| − |
| |
| − |
| |
| − | An important property of the mean is that it includes every value in your data set as part of the calculation. In addition, the mean is the only measure of central tendency where the sum of the deviations of each value from the mean is always zero.
| |
| − |
| |
| − | <br />
| |
| − |
| |
| − | ==== When not to use the mean ====
| |
| − | The mean has one main disadvantage: it is particularly susceptible to the influence of outliers. These are values that are unusual compared to the rest of the data set by being especially small or large in numerical value. For example, consider the wages of staff at a factory below:
| |
| − | {| class="wikitable"
| |
| − | !Staff
| |
| − | !1
| |
| − | !2
| |
| − | !3
| |
| − | !4
| |
| − | !5
| |
| − | !6
| |
| − | !7
| |
| − | !8
| |
| − | !9
| |
| − | !10
| |
| − | |-
| |
| − | |'''Salary'''
| |
| − | |15k
| |
| − | |18k
| |
| − | |16k
| |
| − | |14k
| |
| − | |15k
| |
| − | |15k
| |
| − | |12k
| |
| − | |17k
| |
| − | |90k
| |
| − | |95k
| |
| − | |}
| |
| − | The mean salary for these ten staff is $30.7k. However, inspecting the raw data suggests that this mean value might not be the best way to accurately reflect the typical salary of a worker, as most workers have salaries in the $12k to 18k range. The mean is being skewed by the two large salaries. Therefore, in this situation, we would like to have a better measure of central tendency. As we will find out later, taking the median would be a better measure of central tendency in this situation.
| |
