Logistic Regression

Author

Anqi Fu and Balasubramanian Narasimhan

Introduction

In classification problems, the goal is to predict the class membership based on predictors. Often there are two classes and one of the most popular methods for binary classification is logistic regression Freedman (2009).

Suppose now that $y_{i} \in {0, 1}$ is a binary class indicator. The conditional response is modeled as $y | x \sim Bernoulli (g_{β} (x))$ , where $g_{β} (x) = \frac{1}{1 + e^{- x^{T} β}}$ is the logistic function, and maximize the log-likelihood function, yielding the optimization problem

$\begin{array}{ll} \underset{β}{maximize} & \sum_{i = 1}^{m} {y_{i} \log (g_{β} (x_{i})) + (1 - y_{i}) \log (1 - g_{β} (x_{i}))} . \end{array}$

CVXR provides the logistic atom as a shortcut for $f (z) = \log (1 + e^{z})$ to express the optimization problem. One may be tempted to use log(1 + exp(X %*% beta)) as in conventional R syntax. However, this representation of $f (z)$ violates the DCP composition rule, so the CVXR parser will reject the problem even though the objective is convex. Users who wish to employ a function that is convex, but not DCP compliant should check the documentation for a custom atom or consider a different formulation.

Example

The formulation is very similar to OLS, except for the specification of the objective.

In the example below, we demonstrate a key feature of CVXR, that of evaluating various functions of the variables that are solutions to the optimization problem. For instance, the log-odds, $X \hat{β}$ , where $\hat{β}$ is the logistic regression estimate, is simply specified as X %*% beta below, and the value() function will compute its value after solving. (Any other function of the estimate can be similarly computed.)

n <- 20
m <- 1000
offset <- 0
sigma <- 45
DENSITY <- 0.2

set.seed(183991)
beta_true <- stats::rnorm(n)
idxs <- sample(n, size = floor((1-DENSITY)*n), replace = FALSE)
beta_true[idxs] <- 0
X <- matrix(stats::rnorm(m*n, 0, 5), nrow = m, ncol = n)
y <- sign(X %*% beta_true + offset + stats::rnorm(m, 0, sigma))

beta <- Variable(n)
obj <- -sum(logistic(-X[y <= 0, ] %*% beta)) - sum(logistic(X[y == 1, ] %*% beta))
prob <- Problem(Maximize(obj))
psolve(prob)

[1] -676.7806

check_solver_status(prob)

log_odds <- value(X %*% beta)
beta_res <- value(beta)
y_probs <- 1/(1 + exp(-X %*% beta_res))

We can compare with the standard stats::glm estimate.

d <- data.frame(y = as.numeric(y > 0), X = X)
glm <- stats::glm(formula = y ~ 0 + X, family = "binomial", data = d)
est.table <- data.frame("CVXR.est" = beta_res, "GLM.est" = coef(glm))
rownames(est.table) <- paste0("\\(\\beta_{", 1:n, "}\\)")
knitr::kable(est.table, format = "html", escape = FALSE) |>
    kable_styling("striped") |>
    column_spec(1:3, background = "#ececec")

	CVXR.est	GLM.est
$β_{1}$	-0.0305474	0.0305494
$β_{2}$	0.0023529	-0.0023528
$β_{3}$	-0.0110074	0.0110080
$β_{4}$	0.0163910	-0.0163919
$β_{5}$	0.0157182	-0.0157186
$β_{6}$	0.0006248	-0.0006251
$β_{7}$	-0.0157905	0.0157914
$β_{8}$	-0.0092225	0.0092228
$β_{9}$	0.0173815	-0.0173823
$β_{10}$	0.0019104	-0.0019102
$β_{11}$	-0.0100738	0.0100746
$β_{12}$	-0.0269868	0.0269883
$β_{13}$	0.0233613	-0.0233625
$β_{14}$	0.0009525	-0.0009529
$β_{15}$	-0.0016261	0.0016264
$β_{16}$	0.0312142	-0.0312156
$β_{17}$	0.0038948	-0.0038949
$β_{18}$	-0.0121101	0.0121105
$β_{19}$	0.0246791	-0.0246811
$β_{20}$	-0.0007024	0.0007025

The sign difference is due to the coding of $y$ as $(- 1, 1)$ for CVXR rather than $(0, 1)$ for stats::glm.

So, for completeness, if we were to code the $y$ as $(0, 1)$ , the objective will have to be modified as below.

obj <- -sum(X[y <= 0, ] %*% beta) - sum(logistic(-X %*% beta))
prob <- Problem(Maximize(obj))
psolve(prob)

[1] -676.7806

check_solver_status(prob)
beta_log <- value(beta)
est.table <- data.frame("CVXR.est" = beta_log, "GLM.est" = coef(glm))
rownames(est.table) <- paste0("\\(\\beta_{", 1:n, "}\\)")
knitr::kable(est.table, format = "html", escape = FALSE) |>
    kable_styling("striped") |>
    column_spec(1:3, background = "#ececec")

	CVXR.est	GLM.est
$β_{1}$	0.0305490	0.0305494
$β_{2}$	-0.0023528	-0.0023528
$β_{3}$	0.0110079	0.0110080
$β_{4}$	-0.0163917	-0.0163919
$β_{5}$	-0.0157185	-0.0157186
$β_{6}$	-0.0006250	-0.0006251
$β_{7}$	0.0157912	0.0157914
$β_{8}$	0.0092228	0.0092228
$β_{9}$	-0.0173821	-0.0173823
$β_{10}$	-0.0019103	-0.0019102
$β_{11}$	0.0100744	0.0100746
$β_{12}$	0.0269880	0.0269883
$β_{13}$	-0.0233623	-0.0233625
$β_{14}$	-0.0009528	-0.0009529
$β_{15}$	0.0016264	0.0016264
$β_{16}$	-0.0312154	-0.0312156
$β_{17}$	-0.0038949	-0.0038949
$β_{18}$	0.0121105	0.0121105
$β_{19}$	-0.0246807	-0.0246811
$β_{20}$	0.0007025	0.0007025

Now, the results match perfectly.

Session Info

R version 4.5.2 (2025-10-31)
Platform: aarch64-apple-darwin20
Running under: macOS Tahoe 26.3

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRblas.0.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] kableExtra_1.4.0 CVXR_1.8.0.9207 

loaded via a namespace (and not attached):
 [1] gmp_0.7-5.1        clarabel_0.11.2    xml2_1.5.2         slam_0.1-55       
 [5] stringi_1.8.7      lattice_0.22-9     digest_0.6.39      magrittr_2.0.4    
 [9] evaluate_1.0.5     grid_4.5.2         RColorBrewer_1.1-3 fastmap_1.2.0     
[13] rprojroot_2.1.1    jsonlite_2.0.0     Matrix_1.7-4       ECOSolveR_0.6.1   
[17] backports_1.5.0    scs_3.2.7          Rmosek_11.1.1      viridisLite_0.4.3 
[21] scales_1.4.0       codetools_0.2-20   textshaping_1.0.4  cli_3.6.5         
[25] rlang_1.1.7        Rglpk_0.6-5.1      yaml_2.3.12        otel_0.2.0        
[29] tools_4.5.2        osqp_1.0.0         Rcplex_0.3-8       checkmate_2.3.4   
[33] here_1.0.2         gurobi_13.0-1      vctrs_0.7.1        R6_2.6.1          
[37] lifecycle_1.0.5    stringr_1.6.0      htmlwidgets_1.6.4  cccp_0.3-3        
[41] glue_1.8.0         Rcpp_1.1.1         systemfonts_1.3.1  xfun_0.56         
[45] rstudioapi_0.18.0  knitr_1.51         dichromat_2.0-0.1  highs_1.12.0-3    
[49] farver_2.1.2       htmltools_0.5.9    rmarkdown_2.30     svglite_2.2.2     
[53] piqp_0.6.2         compiler_4.5.2     S7_0.2.1

References

Cox, D. R. 1958. “The Regression Analysis of Binary Sequences.” Journal of the Royal Statistical Society. Series B (Methodological) 20 (2): 215–42.

Freedman, D. A. 2009. Statistical Models: Theory and Practice. Cambridge University Press.