Residuals from model fit as segments

stat_fit_deviations fits a model and returns fitted values and residuals ready for highlighting them as segments in a y vs. x plot.

Usage

stat_fit_deviations(
  mapping = NULL,
  data = NULL,
  geom = "segment",
  position = "identity",
  ...,
  orientation = NA,
  method = "lm",
  method.args = list(),
  n.min = 2L,
  formula = NULL,
  fit.seed = NA,
  na.rm = FALSE,
  show.legend = TRUE,
  inherit.aes = TRUE
)

stat_fit_fitted(
  mapping = NULL,
  data = NULL,
  geom = "point",
  position = "identity",
  orientation = NA,
  ...,
  method = "lm",
  method.args = list(),
  n.min = 2L,
  formula = NULL,
  fit.seed = NA,
  na.rm = FALSE,
  show.legend = FALSE,
  inherit.aes = TRUE
)

Arguments

mapping: The aesthetic mapping, usually constructed with aes(). Only needs to be set at the layer level if you are overriding the plot defaults.
data: A layer specific dataset, only needed if you want to override the plot defaults.
geom: The geometric object to use display the data
position: The position adjustment to use for overlapping points on this layer.
...: other arguments passed on to layer. This can include aesthetics whose values you want to set, not map. See layer for more details.
orientation: character Either "x" or "y" controlling the default for formula. The letter indicates the aesthetic considered the explanatory variable in the model fit.
method: function or character If character, "lm", "rlm", "lmrob", "lts", "gls", "ma", "sma", "segreg", "rq" or the name of a model fit function are accepted, possibly followed by the fit function's method argument separated by a colon (e.g. "rlm:M"). If a function is different to lm(), rlm(), ltsReg(), gls(), ma, sma, it must have formal parameters named formula, data, and weights. See Details.
method.args: named list with additional arguments. Not data or weights which are always passed through aesthetic mappings.
n.min: integer Minimum number of distinct values in the explanatory variable (on the rhs of formula) for fitting to the attempted.
formula: a formula object. Using aesthetic names x and y instead of original variable names.
fit.seed: RNG seed argument passed to set.seed(). Defaults to NA, indicating that set.seed() should not be called.
na.rm: a logical indicating whether NA values should be stripped before the computation proceeds.
show.legend: logical. Should this layer be included in the legends? NA, the default, includes if any aesthetics are mapped. FALSE never includes, and TRUE always includes.
inherit.aes: If FALSE, overrides the default aesthetics, rather than combining with them. This is most useful for helper functions that define both data and aesthetics and shouldn't inherit behaviour from the default plot specification, e.g. borders.

Details

stat_fit_deviations() can be used to automatically highlight residuals as segments in a plot of a fitted model equation. This stat only returns the fitted values and observations, the prediction and its confidence need to be separately added to the plot when desired. Thus, to make sure that the same model formula is used in all plot layers, it is best to save the formula as an object and supply this object as argument to the different statistics.

stat_fit_fitted() only returns the original x and the fitted values, by default mapped to y and can be used to add them to a plot. e.g., as points. Fitted values are available for all model fit methods.

Note

In the case of method = "rq" quantiles are fixed at tau = 0.5 unless method.args has length > 0. Parameter orientation is redundant as it only affects the default for formula but is included for consistency with ggplot2.

Computed variables

Data frame with same nrow as data as subset for each group containing five numeric variables.

x: x coordinates of observations
x.fitted: x coordinates of fitted values
y: y coordinates of observations
y.fitted: y coordinates of fitted values
weights: the weights passed as input to lm(), rlm(), or lmrob(), using aesthetic weight. More generally the value returned by weights()
robustness.weights: the "weights" of the applied minimization criterion relative to those of OLS in rlm(), or lmrob()

To explore the values returned by this statistic we suggest the use of geom_debug(). An example is shown below, where one can also see in addition to the computed values the default mapping of the fitted values to aesthetics xend and yend.

Prior and posterior weights

Two types of weights are possible: prior ones supplied in the call, and posterior weights (called "robustness weights" in robust regression methods) implicitly or explicitly used by fit methods to address heterogeneity of error variance, including the presence of outlier observations . Not all the supported methods accepts prior weights and gls() returns posterior weights that are not in 0..1 like in the case of most other fits. When not accessible weights are set to 1 when known to be equal to 1, which is the most frequent case, or to NA otherwise.

How weights are applied to residuals depends on the method used to fit the model. For ordinary least squares (OLS), weights are applied to the squares of the residuals, so the weighted residuals are obtained by multiplying the "deviance" residuals by the square root of the weights. When residuals are penalized differently to fit a model, the weighted residuals need to be computed accordingly.

Model formula and model fitting

A ggplot statistic receives as data a data frame that is not the one passed as argument by the user, but instead a data frame with the variables mapped to aesthetics. In stat_poly_eq() the compute function is applied by group, each call "seeing" the subset of data for an individual group. As supported models are for regression lines, variables mapped to x and y should both be continuous, i.e., numeric or date time and model formulas defined using x and y as variable names.

The interpretation of the argument passed to formula is enhanced compared to stat_smooth(). Formulas with x as explanatory variable work as in stat_smooth() but formulas with y as explanatory variable are also accepted. orientation is set automatically based on which explanatory variable appears in the formula. Spline-based smoothers are only partially supported.

Model fit methods supported

Several model fit functions are supported explicitly (see tables), and some of their differences smoothed out. Compatibility is checked late, based on the class of the returned fitted model object. This makes it possible to use wrapper functions that do model selection or other adjustments to the fit procedure on a per panel or per group basis. Moreover, if the value returned as model fit object is NULL, plotting is skipped on a per group within panel basis.

In the case of fitted model objects of classes not explicitly supported an attempt is made to find the usual accessors and/or fitted object members, and if found, either complete or partial support is frequently achieved. In this case a message is issued encouraging users to check the validity of the values extracted.

The argument to parameter method can be either the name of a function object, possibly using double colon notation in case its package is not attached, or a character string matching the function name for functions in the search path. This approach makes it possible to support model fit functions that are not dependencies of 'ggpmisc'. Either by attaching the package where the function is defined and passing it by name or as string, or using double colon notation when passing the name of the function.

User-defined functions can be passed as argument to parameter method as long as they have parameters formula, data subset and possibly weights. Additional arguments can be passed to any method as a named list as argument to parameter method.args. As in stat_smooth() prior weights are passed to the model fit functions' weights (plural!) parameter by mapping a numeric variable to plot aesthetic weight (singular!).

Tables 1 lists natively supported model fit functions, with the caveat that only some 'broom' methods' specializations have been actually tested with statistics from 'ggpmisc'. In addition, the statistics based on 'broom' methods require the user to tailor their behaviour by passing additional arguments in the call and occasionally some detective work to find out the names of variables in the returned data frame.

Table 1. Model fit methods supported by the different statistics available in package 'ggpmisc'. Column \(f\) indicates whether computations are done by group (G) or by plot panel (P).

Statistic	\(f\)	Supported model fit methods
`stat_poly_line()`	G	"lm", "rlm", "lts", "sma", "ma", "gls", others with methods `predict()` or `fitted()`
`stat_poly_eq()`	G	"lm", "rlm", "lts", "sma", "ma", "gls", others with needed accesors
`stat_quant_line()`	G	"rq", "rqss"
`stat_quant_band()`	G	"rq", "rqss"
`stat_quant_eq()`	G	"rq", "rqss"
`stat_ma_line()`	G	"SMA", "MA", "RMA", "OLS"
`stat_ma_eq()`	G	"SMA", "MA", "RMA", "OLS"
`stat_fit_residuals()`	G	"lm", "rlm", "lts", "sma", "ma", "gls", "rq", "rqss" others with method `residuals()`
`stat_fit_fitted()`	G	"lm", "rlm", "lts", "gls", "rq", "rqss" others with method `fitted()`
`stat_fit_deviations()`	G	"lm", "rlm", "lts", "gls", "rq", "rqss" others with methods `fitted()` and `weights()`
`stat_fit_augment()`	G	any with 'broom' method `augment()`
`stat_fit_glance()`	G	any with 'broom' method `glance()`
`stat_fit_tidy()`	G	any with 'broom' method `tidy()`
`stat_fit_tb()`	P	any with 'broom' method `tidy()`

The single colon notation is based on parsing the name and is available whenever passing the name of the fit method as a character string. In a string such as "head:tail" the "head" gives the name of the model fit function and the "tail" gives the argument to pass it's method parameter. This is only a convenience, as method.args can be also used. In some methods, i.e., splines, the default formula = y ~ x needs to be overridden with an explicit argument.

Table 2 lists the correspondence of pre-defined method names to model fit method functions. As mentioned above, these are only a subset of the model fit methods that are expected to work. When using these names there is no need for users to attach additional packages but the packages must be available (installed).

Table 2. Available predefined method names, the model fit functions they call, the packages where the functions reside, the class of the returned fitted model object and the arguments that can be passed to their method parameter using single colon notation.

Predefined method names	Model fit methods	R package	Object class
"lm", "lm:qr"	`lm()`	'stats'	"lm"
"rlm", "rlm:M", "rlm:MM"	`rlm()`	'MASS'	"rlm" ("lm")
"lts", "ltsReg"	`ltsReg()`	'robustbase'	"lts"
"ma", "sma", "sma:SMA", "sma:MA", "sma:OLS"	`sma()`	'smatr'	"ma" or "sma"
"gls", "gls:REML", "gls:ML"	`gls()`	'nlme'	"gls"
"rq", "rq:sfn", "rq:sfnc", "rq:lasso"	`rq()`	'quantreg'	"rq"
"rqss", "rqss:sfn", "rqss:sfnc", "rqss:lasso"	`rqss()`	'quantreg'	"rqss"
"SMA", "MA", "RMA", "OLS"	`lmodel2()`	'lmodel2'	("list")

Aesthetics

stat_fit_deviations() understands the following aesthetics. Required aesthetics are displayed in bold and defaults are displayed for optional aesthetics:

•	`x`
•	`y`
•	`group`	→ inferred
•	`xend`	→ `after_stat(x.fitted)`
•	`yend`	→ `after_stat(y.fitted)`

stat_fit_fitted() understands the following aesthetics. Required aesthetics are displayed in bold and defaults are displayed for optional aesthetics:

•	`x`
•	`y`
•	`group`	→ inferred

Learn more about setting these aesthetics in vignette("ggplot2-specs").

Examples

# generate artificial data
library(MASS)

set.seed(4321)
x <- 1:100
y <- (x + x^2 + x^3) + rnorm(length(x), mean = 0, sd = mean(x^3) / 4)
my.data <- data.frame(x, y)

# plot residuals from linear model
ggplot(my.data, aes(x, y)) +
  stat_poly_line(method = "lm", formula = y ~ x) +
  stat_fit_deviations(method = "lm", formula = y ~ x, colour = "red") +
  geom_point()


# plot residuals from linear model with y as explanatory variable
ggplot(my.data, aes(x, y)) +
  stat_poly_line(method = "lm", formula = x ~ y) +
  stat_fit_deviations(method = "lm", formula = x ~ y, colour = "red") +
  geom_point()


# both regressions and their deviations
ggplot(my.data, aes(x, y)) +
  stat_poly_line(method = "lm", formula = y ~ x) +
  stat_fit_deviations(method = "lm", formula = y ~ x, colour = "red") +
  stat_poly_line(method = "lm", formula = x ~ y) +
  stat_fit_deviations(method = "lm", formula = x ~ y, colour = "orange") +
  geom_point()


# give a name to a formula
my.formula <- y ~ poly(x, 3, raw = TRUE)

# plot linear regression
ggplot(my.data, aes(x, y)) +
  stat_poly_line(method = "lm", formula = my.formula) +
  stat_fit_deviations(formula = my.formula, colour = "red") +
  geom_point()


ggplot(my.data, aes(x, y)) +
  stat_poly_line(formula = my.formula, method = "lm") +
  stat_fit_deviations(formula = my.formula, method = "lm", colour = "red") +
  geom_point()


# plot robust regression
ggplot(my.data, aes(x, y)) +
  stat_poly_line(formula = my.formula, method = "rlm") +
  stat_fit_deviations(formula = my.formula, method = "rlm", colour = "red") +
  geom_point()


# plot robust regression with weights indicated by colour
my.data.outlier <- my.data
my.data.outlier[6, "y"] <- my.data.outlier[6, "y"] * 10
ggplot(my.data.outlier, aes(x, y)) +
  stat_poly_line(method = MASS::rlm, formula = my.formula) +
  stat_fit_deviations(formula = my.formula, method = "rlm",
                      mapping = aes(colour = after_stat(robustness.weights)),
                      show.legend = TRUE) +
  scale_color_gradient(low = "red", high = "blue", limits = c(0, 1),
                       guide = "colourbar") +
  geom_point()


# plot quantile regression (= median regression)
ggplot(my.data, aes(x, y)) +
  stat_quantile(formula = my.formula, quantiles = 0.5) +
  stat_fit_deviations(formula = my.formula, method = "rq", colour = "red") +
  geom_point()


# plot quantile regression (= "quartile" regression)
ggplot(my.data, aes(x, y)) +
  stat_quantile(formula = my.formula, quantiles = 0.75) +
  stat_fit_deviations(formula = my.formula, colour = "red",
                      method = "rq", method.args = list(tau = 0.75)) +
  geom_point()


# inspecting the returned data with geom_debug_group()
gginnards.installed <- requireNamespace("gginnards", quietly = TRUE)

if (gginnards.installed)
  library(gginnards)

# plot, using geom_debug_group() to explore the after_stat data
if (gginnards.installed)
  ggplot(my.data, aes(x, y)) +
    stat_poly_line(method = "lm", formula = my.formula) +
    stat_fit_deviations(formula = my.formula,
                        geom = "debug_group") +
    geom_point()

#> [1] "PANEL 1; group(s) -1; 'draw_function()' input 'data' (head):"
#>   x          y x.fitted   y.fitted weights robustness.weights hjust PANEL group
#> 1 1 -27205.450        1 -3691.7638       1                  1     0     1    -1
#> 2 2 -14242.651        2 -2578.9186       1                  1     0     1    -1
#> 3 3  45790.918        3 -1498.5419       1                  1     0     1    -1
#> 4 4  53731.420        4  -443.8109       1                  1     0     1    -1
#> 5 5  -8028.578        5   592.0973       1                  1     0     1    -1
#> 6 6 102863.943        6  1616.0056       1                  1     0     1    -1
#>   xend       yend orientation
#> 1    1 -3691.7638           x
#> 2    2 -2578.9186           x
#> 3    3 -1498.5419           x
#> 4    4  -443.8109           x
#> 5    5   592.0973           x
#> 6    6  1616.0056           x

if (gginnards.installed)
  ggplot(my.data.outlier, aes(x, y)) +
    stat_poly_line(method = "rlm", formula = my.formula) +
    stat_fit_deviations(formula = my.formula, method = "rlm",
                        geom = "debug_group") +
    geom_point()

#> [1] "PANEL 1; group(s) -1; 'draw_function()' input 'data' (head):"
#>   x           y x.fitted   y.fitted weights robustness.weights hjust PANEL
#> 1 1  -27205.450        1 -2515.4775       1         1.00000000     0     1
#> 2 2  -14242.651        2 -1531.5327       1         1.00000000     0     1
#> 3 3   45790.918        3  -573.5792       1         1.00000000     0     1
#> 4 4   53731.420        4   365.1016       1         1.00000000     0     1
#> 5 5   -8028.578        5  1291.2282       1         1.00000000     0     1
#> 6 6 1028639.429        6  2211.5193       1         0.07650433     0     1
#>   group xend       yend orientation
#> 1    -1    1 -2515.4775           x
#> 2    -1    2 -1531.5327           x
#> 3    -1    3  -573.5792           x
#> 4    -1    4   365.1016           x
#> 5    -1    5  1291.2282           x
#> 6    -1    6  2211.5193           x