Statistics and Data Analysis
Least Trimmed Squares Regression
Correlogram
Detrend
Difference
More Updates
The LeastTrimmedSquares command computes least trimmed squares regression for some data.
withStatistics:
In this example, we have 1000 data points. There is a single independent variable, x, with values uniformly distributed between 0 and 10. The dependent variable is a linear function of the independent variable plus some additive noise, y=5 x + 10 + noise, where the noise is from a probability distribution known to have severe outliers - the Cauchy distribution, with location parameter 0 and scale parameter 5.
x≔SampleUniform0,10,1000:
noise≔SampleCauchy0,1,1000:
y ≔ 5*x + noise +~ 10:
Here we see all data points:
pp ≔ PointPloty, xcoords = x,size=0.7 , golden: pp;
Linear least squares regression will be severely affected by the outliers.
ls_regression_result ≔ Fita * X + b, x, y, X;
ls_regression_result≔3.44970682383807X+10.6816568681413
ls_deviation_from_model ≔ coeffls_regression_result, X, 1 − 52 + coeffls_regression_result, X, 0 − 102;
ls_deviation_from_model≔2.86806501793850
Least trimmed squares regression gets much closer to the true line without noise.
lts_regression_result ≔ LeastTrimmedSquaresx, y, X;
lts_regression_result≔5.03537530551575X+9.82475419561272
lts_deviation_from_model ≔ coefflts_regression_result, X, 1 − 52 + coefflts_regression_result, X, 0 − 102;
lts_deviation_from_model≔0.0319625041956780
The result is even better if we include 900 out of the 1000 points, instead of the default of a little over 500.
lts_900_regression_result ≔ LeastTrimmedSquaresx, y, X, include=900;
lts_900_regression_result≔5.00862730339998X+10.0156318668695
lts_900_deviation_from_model ≔ coefflts_900_regression_result, X, 1 − 52 + coefflts_900_regression_result, X, 0 − 102;
lts_900_deviation_from_model≔0.000318785625780919
The other robust regression method, implemented in the RepeatedMedianEstimator procedure, also gets a good result.
rme_regression_result ≔ RepeatedMedianEstimatorx, y, X;
rme_regression_result≔10.0306661686300+5.00564125476873X
rme_deviation_from_model ≔ coeffrme_regression_result, X, 1 − 52+ coeffrme_regression_result, X, 0 − 102;
rme_deviation_from_model≔0.000972237653807886
In order to visualize these results, we show the same point plot as before, including the four regression lines. The three regression lines from robust methods cannot be distinguished, but the least squares method is clearly off. We zoom in on the vertical range that includes most points.
plotsdisplaypp, plotls_regression_result, lts_regression_result, lts_900_regression_result, rme_regression_result, X=0..10, legend=Least squares, Least trimmed squares, Least trimmed squares (900 points), Repeated median estimator, view=0..10, −10..110,size=0.7 , golden;
The Correlogram command computes autocorrelations of a data set and displays the result as a column plot with dashed lines indicating the lower and upper 95% confidence bands for the normal distribution N(0,1/L), where L is the size of the sample 'X', and a caption reporting how many of the displayed columns lie outside of the bands of plus or minus 2, 3, and 4 standard deviations respectively. AutoCorrelationPlot is an alias for the Correlogram command.
CorrelogramImportdatasets/sunspots.csv,base=datadir,output=Matrix265..310, 2
The Detrend command removes any trend from a set of data.
restart:
For example, specify some data:
data≔Matrix0,1.8,1,0.7,2.5,2.8,4,4.2,6.2,3
data≔01.810.72.52.844.26.23
Fit a linear model to the data:
lm≔LinearFita+bt,data,t
lm≔1.49598376946009+0.366429281218947t
It can be observed that from the plot of the data and the linear model that there is some upward trend. The Detrend command removes any trend from the data.
detrend_data≔Detrenddata
detrend_data≔0.304016230539914−1.162413050679030.3879430274925471.23829910566413−0.767845313017555
This can be observed in the following plot:
plots:-display ScatterPlotdata,color=Black,legend=Original Data,symbol=solidcircle,symbolsize=15, plotlm,color=Black,legend=Trend,linestyle=dash, ScatterPlotdata..,1,detrend_data,color=Red,legend=Detrended Data,symbol=diamond,symbolsize=15, plotMeandetrend_data,color=Red,legend=Mean of Detrended Data,linestyle=dot,view=mindata..,1−0.1..maxdata..,1+0.1,default,size=0.5 , golden
Detrend has also been added as an option to several routines in SignalProcessing including SignalPlot, Periodogram, and Spectrogram.
The Difference command computes lagged differences between elements in a data set.
Define some data:
x≔seqi2,i=1..10
x≔149162536496481100
Differencex
35791113151719
Display of Data Structures
The default display of data structures such as rtables (matrices, vectors, and arrays) as well as data frames and data series has been changed. Previously a summary would be shown for the data structure, but now the display shows the header of the data (by default the first 10 rows and 10 columns) and the size of the data structure.
Matrix(20,20,rand(1..10));
For more details, see the User Interface updates page.
DataFrames and DataSeries
Several commands have been updated to support DataFrames and DataSeries, including remove, select and selectremove. Other new commands such as Detrend and Difference also support DataFrames and DataSeries.
Biplot
The Biplot command has a new option, components, which specifies the principal components used in the biplot.
DataSummary, FivePointSummary, FrequencyTable
The DataSummary, FivePointSummary, and FrequencyTable commands have a new option, tableweights, which specifies the relative column widths in the displayed embedded table.
Download Help Document
