We present a finite element model updating technique for soil-structure system identification of the Millikan Library building using the seismic data recorded during the 2002 Yorba Linda earthquake. A detailed finite element (FE) model of the Millikan Library building is developed in OpenSees and updated using a sequential Bayesian estimation approach for joint parameter and input identification. A two-step system identification approach is devised. First, the fixed-base structural model is updated to estimate the structural model parameters (including effective elastic modulus of structural components, distributed floor mass, and Rayleigh damping parameters) and some uncertain components of the foundation-level motion. Then, the identified structural model is used for soil-structure model updating wherein the Rayleigh damping parameters, the stiffness and viscosity of the soil subsystem (modeled using a substructure approach), and the foundation input motions (FIMs) are estimated. The identified model parameters are compared with state-of-practice recommendations. While a specific application is made for the Millikan Library, the present work offers a framework for integrating large-scale FE models with measurement data for model inversion. By utilizing this framework for different civil structures and earthquake records, key structural model parameters can be estimated from the real-world recorded data, which can subsequently be used for assessing and improving, as necessary, state-of-the-art seismic analysis and structural modeling techniques. This paper presents an effort towards using real-world measurements for large-scale FE model updating in the soil and structure, uniform soil time domain for joint parameter and input estimation, and thus paves the way for future applications in system identification, health monitoring and diagnosis of civil structures.