Nested cross-validation
Nested cross-validation is said to be an improvement over cross-validation.
Unfortunately, I found most explanations quite confusing, so decided to simulate some data and see what happens.
In this post, I simulate two models: one linear model which perfectly fits the data and one which overfits the data.
Next, cross-validation and nested cross-validation are plotted.
To keep the post short, I've hidden the code to produce the plots.
import MLJLinearModels
import MLJDecisionTreeInterface
using DataFrames: DataFrame, select, Not
using Distributions: Normal
using CairoMakie: Axis, Figure, lines, lines!, scatter, scatter!, current_figure, axislegend, help, linkxaxes!, linkyaxes!, xlabel!, density, density!, hidedecorations!, violin!, boxplot!, hidexdecorations!, hideydecorations!
using MLJ: CV, evaluate, models, matching, @load, machine, fit!, predict, predict_mode, rms
using Random: seed!
using Statistics: mean, std, var, median
using MLJTuning: TunedModel, Explicit
using MLJModelInterface: Probabilistic, Deterministic
Show more