Statistics with DataFrames
A DataFrame is one of the basic data structures in Maple. Data frames are a list of variables, known as DataSeries, which are displayed in a rectangular grid. Every column (variable) in a DataFrame has the same length; however, each variable can have a different type, such as integer, float, string, name, truefalse, etc., which makes data frames an ideal storage device for heterogeneous data.
When printed, Data frames resemble matrices in that they are viewed as a rectangular grid, but a key difference is that the first row corresponds to the column (variable) names and the first column corresponds to the row (individual) names. These row and columns are treated as header meta-information and are not a part of the data. Moreover, the data stored in a DataFrame can be accessed using these header names, as well as by the standard numbered index.
This example page shows a number of common statistical operations with data frames. For more information, including a list of Statistics commands that can be run on data frames, see DataFrames in Statistics.
Getting Started
To begin, the Statistics package is loaded.
with(Statistics):
A new DataFrame can be created based on randomly generated data. The following generates a DataFrame with 3 DataSeries. The first two columns are generated by randomly sampling from a Uniform(0,1) distribution. The third DataSeries also has randomly generated data, which is sampled from 4 distinct levels: 0, 1, 2, 3.
data := DataFrame( < Sample(Uniform(0, 1), [20, 2]) | LinearAlgebra:-RandomVector(20, generator = rand(0 .. 3) ) >, columns = [a,b,c] );
DataFrames and the Context Panel
The Maple programming language provides many commands that are useful for exploring DataFrames. The Context Panel provides easy access to a selection of these commands, displaying context-specific commands that can be applied to DataFrames or DataSeries.
The DataFrame context-sensitive options include many commands that can be applied to entire DataFrames as well as to a single DataSeries in a DataFrame. The second section of the DataFrame menu in the Context Panel includes commands for conversions, operations, queries, and visualization of DataFrames and DataSeries. The third section includes more commands relating to statistics and data analysis, including data analysis, data manipulation, properties and quantities, and summary and tabulation.
A useful feature of the context-sensitive options is the ability to quickly filter the DataFrame by value or to select columns to apply operations to. This can be beneficial when dealing with heterogeneous data that includes non-numeric DataSeries. In many cases, commands in Statistics assumes an entirely numeric DataFrame. Selectively removing the non-numeric data makes it possible to use the routines natively on the given DataFrame.
Computing Summary Statistics
One of the most fundamental statistical tasks is to generate summary statistics, such as the mean or standard deviation, for a given data set. The DataSummary command returns the mean, standard deviation, skewness and kurtosis, as well as the minimum and maximum values, and the cumulative weight. When all of the elements in the DataSeries have weight = 1, the cumulative weight corresponds to the total number of observations in the DataSeries.
DataSummary(data, summarize = embed):
a
b
c
mean
0.642033244375975753
0.492561671718578453
1.64999999999999969
standarddeviation
0.329931930346065327
0.334466425079465335
1.26802789276975481
skewness
−0.535097906770228615
−0.223319664925291123
−0.259820705584549505
kurtosis
1.62899290839617339
1.41288760741544195
1.38747927386886905
minimum
0.0975404049994095246
0.0318328463774206760
0.
maximum
0.970592781760615697
0.950222048838354927
3.
cumulativeweight
20.
Individual summary statistics can also be returned. For example, the variance of each of the DataSeries in the DataFrame is:
Variance(data);
It is important to note that a DataSeries is returned from requesting any individual summary statistic. Each row corresponds to each of the columns in the DataFrame. In this case, column "a" has a variance of 0.11, column "b" has a variance of 0.11, and column "c" has a variance of 1.61.
Aggregate Statistics
The defined DataFrame has a third column which has 3 distinct levels: 0, 1, 2, and 3. The distinct levels in a DataSeries can easily be seen by collapsing the DataSeries into a set:
convert(data[3], set);
0,1,2,3
When a DataFrame contains a DataSeries that has a fixed number of distinct levels, it can be useful to compute aggregate statistics based on the levels in a given DataSeries. Using the example DataFrame, since the column "c" has 4 distinct levels, aggregate statistics can be computed using the Aggregate command, that show values for a given statistic for each of the levels.
The following shows the mean for each of the levels in column "c":
Aggregate(data, 3);
The default summary statistic for the Aggregate command is the mean. By specifying a function, any summary statistic can be returned for the levels in column "c":
Aggregate(data, 3, function = Median);
The tally option can be optionally specified in order to tally up the number of observations in each of the levels:
Aggregate(data, 3, function = Range, tally);
Data Manipulation
The Statistics package has many commands for manipulating statistical data including data selection, scaling and more. For example, the Scale command can be used to center and scale numeric DataSeries:
data[[1, 2]] := Scale(data[[1, 2]]);
data;
Data Analysis
There are many commands in Maple for analyzing data, including commands for regression analysis, principal component analysis, analysis of variance and more.
Principal Component Analysis (PCA) aims to identify patterns in data by reducing the dimensionality of multivariate data to a few key explanatory variables called principal components. Performing a principal component analysis on our data returns a record, which can be queried for the principal components, transformation matrix, singular values and more.
pca := PCA(data, summarize = true);
summary:
Values proportion of variance St. Deviation 1.7667 0.4897 1.3292 1.5158 0.4201 1.2312 0.3254 0.0902 0.5704
More information on PCA can be captured using a biplot:
Biplot(pca, pointlabels = true, points = false);
Visualizing DataFrames
Many statistical visualizations support DataFrames and DataSeries:
BoxPlot(data);
AreaChart(data);
The colorscheme option is useful when a DataSeries has a fixed number of distinct levels in a given column. In the following, the points are colored based on the 4 levels {0, 1, 2, 3} found in the column "c".
ScatterPlot(data[.., 1], data[.., 2], symbolsize = 20, symbol = solidbox, colorscheme = ["valuesplit", data[.., 3], [0 = "Red", 1 = "Blue", 2 = "Green", 3 = "Purple"]]);
Sampling a DataFrame
A typical way of sampling a dataset stored in a Matrix is to make a new empirical distribution based on the dataset and sample it directly. The same process can be used for numeric DataSeries:
Sample(EmpiricalDistribution(data[1]), 10);
This method is however only valid for type numeric DataSeries. In the case that we have a non-numeric OR numeric DataSeries, the RandomTools[Generate] command can be used to sample the DataSeries. This does however require a conversion of the DataSeries to a list:
RandomTools:-Generate('list'('choose'(convert(data[3], list)), 10));
0,1,0,0,1,3,3,0,3,3
RandomTools:-Generate('list'('choose'(convert(DataSeries(["m", "a", "p", "l", "e"]), list)), 10));
e,e,l,p,p,e,p,p,e,p
More examples
There are several examples for working with DataFrames and DataSeries:
DataFrame Guide : Examples of working with DataFrames
Iris Data : Examples of summarizing data, computing aggregate statistics, and principal component analysis
Subsets of DataFrames : Examples of indexing and filtering columns and rows of a DataFrame
Download Help Document
