Fast algorithms for elliptic PDEs are central to modern scientific computing. In this talk, we consider the efficient factorization of matrices associated with elliptic problems in both integral and differential form. A key starting point is the nested dissection multifrontal method for PDEs, which can be viewed as an LU factorization with a cost which grows with the spatial dimension. Our primary contributions are twofold: (1) a reformulation of previous fast direct solvers for integral equations as multifrontal-like generalized LU decompositions; and (2) a recursive dimensional reduction strategy to achieve optimal linear or nearly linear complexity in 2D and 3D. Our method is fully adaptive and can handle both boundary and volume problems, and furthermore reveals the close connection between structured dense matrices and sparse ones. This is joint work with Lexing Ying.