Prerpints.org logo
Preprint
Article

Evaluation of Public ICT-based Real Estate Development Projects and Analysis of Operational Performance of Related Companies: Focusing on Smart City Projects in Korea

Altmetrics

Downloads

104

Views

16

Comments

0

This version is not peer-reviewed

Submitted:

25 August 2023

Posted:

29 August 2023

You are already at the latest version

Alerts
Abstract
This study identified, analysed, and prioritised significant factors for standardising the Korean smart city project evaluation system. We analysed the efficiency and productivity of companies currently providing smart city services to consider both policy and practical aspects. The prioritisation of smart city planning reflects the latest trends in South Korea, where urban planning is moving towards smart city planning. Furthermore, the capacity building of public-private/private-private partnerships indicate the importance of business scalability. This indicates that smart city services are only stabilised when the private sector is involved and leads the project, rather than focusing on public development. The feasibility of building intelligent facilities indicates that smart city projects should be implemented after securing cost-benefit feasibility. The results were used as the basis for building an evaluation system, showing that in smart city convergence alliances, small- and medium-sized enterprises achieved the highest efficiency by reducing inputs to 81% and 86%, under the assumptions of constant and variable returns to scales, respectively. As the operational aspect is more problematic than the technical aspect, policy alternatives are necessary for smooth business progress, including increased flexibility of laws and institutions and the activation of policies that temporarily relieve regulations to demonstrate smart city projects.
Keywords: 
Subject: Social Sciences  -   Urban Studies and Planning

1. Introduction

A smart city is based on intelligent infrastructure in which technology is applied to solve existing urban problems and improve citizens’ quality of life [1]. Recently, smart cities are not only being viewed as objects; their non-physical elements are also being emphasised [2], building a foundation to address social problems in collaboration with experts and citizens [3]. The importance and necessity for smart cities is rising because, owing to the COVID-19 pandemic, activities that previously took place in physical spaces have shifted to virtual ones [4]. In recent years, smart cities have been ranked high on the agenda of urban discourse because of significant changes in daily life, including rapid urbanisation rates, a decline of old downtowns, and socio-economic and environmental factors [5]. According to Mckinsey’s market research report, the smart city market is expected to grow up to $1.7 trillion by 2025, and this trend will continue to expand, as most of the countries worldwide, including developing ones, are promoting smart city projects [6]. Although a smart city project is highly important as a national policy, and demonstration projects are actively underway, they are in a relatively early stage, and the actual field application and current-status analysis are often insufficient. Therefore, many are demanding the reorganisation of the public project evaluation system [7]. Given the nature of these projects, which require large amounts of capital, it is necessary to improve their return on investment (ROI); however, no universal performance management system or evaluation indicators are internationally accepted [8]. Deriving standardised evaluation indicators is difficult despite their importance because of the vastness of the concept and comprehensiveness of the project, as well as the lack of adequate performance management systems of companies that actually carry out the project, based on which evaluation factors should be derived [9]. A smart city is a fusion and convergence concept that goes beyond simple urban planning; encompasses technology, economy, society, and politics; and requires cooperation between stakeholders and an organic relationship with the city government. We have been witnessing side effects, including the implementation of multiple simultaneous projects by several government departments and a business model that focuses on building infrastructure to achieve short-term results [10]. Consequently, evaluation indicators vary based on department, project, and participating organisations, and they are not standardised. South Korea operates a smart city convergence alliance, comprising enterprises, universities, research institutes, and associations; however, their operational efficiency is not measured, and they fail to set the future direction and improve the sustainability of the business [11]. In particular, small- and medium-sized enterprises (SMEs) are vulnerable to management risk, based on the impact of the business, and monitoring and consulting on performance is essential. Against this backdrop, this study established an analysis model and derived implications and improvement suggestions for the smart city project evaluation system. First, by reviewing the evaluation system of smart city projects that have been promoted in various forms and collecting opinions through expert focus group interviews (FGIs), we set standard indicators to measure the importance of each evaluation factor and set priorities. Second, we conducted an efficiency and productivity analysis on 38 SMEs, which are most sensitive to business performance among the currently operating participants of the smart city convergence alliance and thus use them as the basis for an evaluation system along with priorities set by evaluation factors. Third, based on the analysis results, we suggested improvements and implications for a future evaluation system and management performance in the operational stage.

1.1. Methodology

This study reviewed the literature on smart city project evaluation systems and reclassified items based on the criteria used in the evaluation. First, we organised the overlapping factors and established the first tiers as technical expertise, plan specificity, sustainability, and scalability. Technical expertise was measured by technology acceptance and administrative accessibility, and plan specificity was divided into economic, urban, and educational infrastructures, with a focus on economic infrastructure. Sustainability, which focuses on social infrastructure, was divided into social, living, and environmental infrastructures, whereas scalability was divided into the cultivation of specialised human resources and the formation of an industrial ecosystem, to derive the second tier. The third tier was selected through five rounds of FGIs to compare the importance of the various types of factors. To standardise the evaluation indicators, they were established as generic indicators and had various implications. The details of the three tiers are discussed in the section on the derivation of the hierarchy diagram. Based on the final hierarchy diagram, we derived priorities based on the item, which were expected to serve as base materials for the establishment of an evaluation system. Next, we measured SMEs’ efficiency and productivity; which are most sensitive to business performance among the participants of the current smart city convergence alliance. Based on the results of the analysis, we provide suggestions and implications to improve a practical evaluation system that combines policy and company operability.

1.2. Scope of the Research

First, a survey was conducted to derive the analytic hierarchy process (AHP) analysis results using the first analysis methodology, that is, the hierarchy diagram. A questionnaire was distributed to 100 experts working on smart cities. A total of 100 questionnaires were collected and 83 valid samples were used, excluding inconsistent responses (17). Survey participants were evenly distributed: 23 university professors, 20 public officials, 17 general workers, 12 researchers, and 11 employees at public corporations. Next, in addition to the qualitative results gathered in the previous analysis, quantitative, efficiency, and productivity analyses were performed to overcome the limitations of the study. The operational status of smart city convergence alliance companies currently in operation was examined. The analysis point was limited to three years from 2019 to 2021, and an analysis model was established to identify changes in trends over the three years. Six variables were used in the analysis, details of which are presented in the results. Ultimately, these results complement the limitations of prior studies by employing both qualitative and quantitative perspectives, which present diverse implications, as the viewpoints of each analysis method coincide.
Table 1. Analysis model design.
Table 1. Analysis model design.
Identification of priority items - Conducted an expert survey based on a finalised hierarchy diagram- Utilised AHP analysis to determine the weight of each item- Identified priorities based on the derived weights
Efficiency and productivity analytics - Measured efficiency and productivity of 38 SMEs in a smart city convergence alliance - Conducted year-by-year analysis based on data from 2019 to 2021- Identified the flow of change through trend analysis rather than a simple point-in-time basis analysis
Analysis results and conclusions - Provided improvements and implications for the future evaluation systems based on the priorities, efficiency, and productivity analysis results

2. Theoretical Considerations

2.1. Smart City Overview

Previous studies on information and cyber cities tended to introduce the smart city concept as the ‘Wired City’ [12] or ‘Digital City’ [13,14]. However, this characterisation has been criticised for being overly technology-driven, focusing on corporate needs, and failing to accommodate urban needs and incorporating social and environmental sustainability [15,16]. In response to this criticism, the Organization for Economic Cooperation and Development (OECD) [17] defines smart cities as ‘initiatives or approaches that effectively leverage digitalisation to boost citizen well-being and deliver more efficient, sustainable, and inclusive urban services and environments as part of a collaborative, multi-stakeholder process’. The International Telecommunication Union (ITU-T) [18] defines a smart city as ‘an innovative city that uses information and communication technologies (ICTs) and other means to improve the quality of life, efficiency of urban operation and services, and competitiveness, while ensuring it meets the needs of present and future generations with respect to economic, social, and environmental aspects’. In the promotion of smart cities, solving urban problems based on digitalisation and ICT has been discussed along with the importance of governance owing to the convergence of ICTs and digitalisation. Consequently, smart cities are evolving to prioritise citizens rather than technology and embrace inclusion, sustainability, and resilience. According to Cohen [19], smart cities will evolve into spaces that incorporate citizens when comprehensively demonstrating various technologies and services; although the introduction and operation of related facilities is essential for providing services using digital infrastructure, it is not sufficient for discovering and providing the services that citizens need. During this process, innovation emerges and spreads, promoting the growth of related industries.
Therefore, the public and private sectors must work together to demonstrate and spread smart cities. A digital infrastructure is needed, and services must be provided based on it. However, even with digital infrastructure, services will not be provided automatically. Smart city policies are being promoted, recognising that various factors such as digital infrastructure and technology, public-private governance system construction, and institutional support are required to provide smart city services [20,21]. In this context, smart city policies in Europe have implications for the spread and distribution of smart cities in South Korea. As a leader in emerging urban issues such as green cities and eco-cities, Europe is also taking the initiative in smart city policies [22]. The European Union’s smart city policy is driven by a program called the European Innovation Partnership on Smart Cities and Communities (EIP-SCC) [23], which comprehensively promotes the demonstration and spread of smart cities, performance evaluation and verification of projects, standardisation, and establishment of a knowledge platform, providing significant implications for smart city promotion policies [24].

2.2. Discussion on Evaluation Indicators for Smart City Projects in South Korea

Smart city projects in Korea are categorised into Smart Challenge, Smart Town, Core Technology Discovery, and Smart Campus Challenge Projects. In this section, the evaluation indicators of each project are reviewed to establish a hierarchy diagram for the AHP analysis. The evaluation indicators for each project can be listed as follows. First, the Smart Challenge Project aims to solve urban problems by utilising creative ideas (solutions) from the private sector, including companies and universities, and spread the best solutions to other local governments and overseas. Unlike past local government support programs, companies, universities (ideas and investments), and local governments (space provision and citizen engagement) were involved in the project from the beginning. Based on these characteristics, smart city project evaluation indicators comprised five written assessment items and two presentation items [25].
Table 2. 2021 Smart Challenge Project evaluation indicators.
Table 2. 2021 Smart Challenge Project evaluation indicators.
Div. Item Score Sub-items
Written assessment (60%) Necessity and feasibility of project 20 - Analysis of the current situation and problems at the target site and feasibility of solutions
- Goals and objectives and the need to implement preliminary and main projects
- Specificity and achievability of quantitative performance goals (KPIs) for problem-solving
- Whether the local government has established (or is establishing) a smart city plan
Excellence and innovation of the preliminary project
* Demonstration plan of key solutions		 30 - Appropriateness and innovativeness of key solutions to achieve goals
- Feasibility of demonstration and adoption of the proposed solution
- Economic and social effects of the proposed solution
- Appropriateness of securing, linking, integrating, and utilising data collected by the solution
- Domestic and international spread of the proposed solution (commercialisation, business model, etc.)
- Post-operation and management plan for solutions and services
Promotion system adequacy Collaboration of citizens & stakeholders 10 - Diversity of tools for citizen engagement and stakeholder conflict control
- Specificity of citizen engagement and stakeholder conflict control measures based on project phase
Interagency collaboration 20 - Uniqueness and excellence of the governance system for preliminary project
- Participation and recruitment plan of private companies
- Financing and investment plans of local governments, companies, etc.
Specificity and excellence of the main plan 20 - Basic direction, promotion strategy, and implementation plan of the main project (phased roadmap, etc.)
- Specificity of sustainable promotion measures such as management and operation
- Specificity of the plan to secure the budget for the main project
Presentation (40%) Commitment to the project implementation Local government commitment and feasibility 35 - Excellence of local government conditions and capacity to promote smart cities
- Appropriateness of project goals and specificity of local government support and strategies
- Feasibility of the local government’s budget and investment plan
- Measures to ensure the sustainability of solution operations and management and service provisions
Private actors’ willingness to commercialise 35 - Appropriateness of plans to engage, recruit, and cooperate with private companies, etc.
- Feasibility of the financing and investment plans of companies in implementing the project
- Domestic and international expansion plans, including commercialisation of the solutions
- Measures to promote sustainable smart cities, including the introduction of business models
Expected effects and sustainability of the project 30 - Innovation, excellence, and sustainability of the proposed key solutions
- Potential to comprehensively solve urban problems through preliminary and main projects
- Feasibility of the project management plan for visible results
- Measures to achieve socio-economic ripple effects such as enhancing regional competitiveness and job creation
Second, the Smart Town Challenge Project was promoted to create region-specific smart towns within medium- and small-sized cities to lay the foundation for the spread of smart solutions to other local governments through demonstrations, while considering local needs and conditions. Based on these characteristics, the evaluation indicators for the Smart Town Project comprised four written assessment items and two presentation items [26].
Table 3. 2021 Smart Town Project evaluation indicators.
Table 3. 2021 Smart Town Project evaluation indicators.
Div. Item Score Sub-items
Written assessment
(60%)		 Necessity and feasibility of the project 20 - Validity of target sites’ status and need analysis
- Specificity and feasibility of vision, goals, and strategies
- Specificity and feasibility of solution-specific KPIs
Excellence of the project plan 30 - Alignment with the project type (theme) and alignment between solutions
- Track record of identifying local demand for the project plan
- Excellence and specificity of the project plan
Adequacy of the project plan 30 - Specificity and excellence of the project implementation organisation and cooperation governance structure and operation plan
- Performance of preliminary consultations with related organisations and departments for construction, operation, and management, as well as specificity of future cooperation plans
- Excellence of project management plan to prevent delays in project implementation
- Specificity of local government financing plan
Sustainability of the project 20 - Specificity of the post-operational management plan
- Possibility of spreading to other local governments and expected effects
Presentation
(40%)		 Commitment to the project
 50 - Commitment to the project and specificity of the implementation plan
- Does the business plan reflect local conditions?
- Excellence and specificity of the business plan
- Feasibility of the local government’s business plan, including securing budget and cooperation with related organisations
- Excellence of the project management plan to prevent delays in project implementation
Expected effects and sustainability of the project 50 - Expected outcomes (solving local problems, spreading to other local governments, etc.)
- Appropriateness and feasibility of quantitative KPIs
- Excellence in post-project operation management plan
- How to secure the sustainability of smart town creations
Third, Core Technology Discovery Projects are divided into ‘company-led’ types, targeting companies or universities with innovative technologies, and ‘citizen-led living lab’ types, led by local citizens, which solve urban problems. First, unlike previous local government support programs, ‘company-led’ projects aim to discover and demonstrate differentiated solutions among companies and universities that own innovative technologies. Administrative and financial support is provided to facilitate the implementation of innovative solutions, along with special procurement incentives such as the designation of innovative products with superior technology. The evaluation indicators for Core Technology Discovery Project comprised four written assessment items and three presentation evaluation items [27].
Table 4. 2022 Core Technology Discovery Project (company-led type) evaluation indicators.
Table 4. 2022 Core Technology Discovery Project (company-led type) evaluation indicators.
Div. Item Score Sub-items
Written assessment (60%) Necessity and feasibility of the project 20 - Analysis of the current situation and problems at the target site and feasibility of solutions
- Project goals and need of implementation
Excellence and innovation of the technology 30 - Adequacy of solutions to achieve goals
- Innovativeness of the technology/solution and its differentiation from existing solutions (economic and technological advantage, innovative utilisation, etc.)
- Feasibility of the demonstration and adoption plan of the proposed solution
- Specificity and achievability of quantitative KPIs
Appropriateness of project direction 30 - Appropriateness of project governance system and collaboration arrangements
- Specificity of citizen participation and stakeholder conflict control plans for each stage
- Appropriateness of securing, linking, integrating, and utilising data collected by the solution
- Specificity of sustainable promotion measures, such as reflecting in smart city planning
Appropriateness of the budget use plan 20 - Specificity and appropriateness of project budget utilisation plans
- Financing and investment plans of companies and local governments (labour cost accounting ratio, etc.)
Presentation (40%) Commitment to the project
implementation		 Private actors’ willingness to commercialise 40 - Feasibility of the financing and investment plans of companies in implementing the project
- Domestic and international expansion plans, including commercialisation of the solutions
- Measures to promote sustainable smart cities, including the introduction of business models
Local government commitment and feasibility 30 - Specificity of local government support and strategies to achieve goals
- Feasibility of the local government’s budget and investment plan
- Measures to ensure the sustainability of solution operations, and management and service provision
Expected effects and sustainability 30 - Innovation, excellence, and sustainability of the proposed key solutions
- Potential to comprehensively solve urban problems through the project
- Economic and social effects of the proposed solution
- Feasibility of project management plan for visible results
- Measures to achieve socio-economic ripple effects such as enhancing regional competitiveness and job creation
‘Citizen-led living lab’ projects support companies and universities with innovative technologies or local governments that aim to demonstrate innovative technology by establishing and operating a sustainable smart city living lab tailored to each region. Companies and universities should clarify their target sites and collaborate with local governments and public institutions to provide administrative and financial support. Based on these characteristics, the evaluation indicators for citizen-led living lab type projects were identified as ‘Commitment to the project implementation’ and ‘Expected effects and sustainability of the project’, based on ‘Necessity and feasibility of the project’, ‘Excellence and innovativeness of the project’, ‘Living lab operation plan’, ‘Budget investment and utilisation plan’, and ‘sustainability’.
Table 5. 2022 Core Technology Discovery Project (citizen-led living lab type) evaluation indicators.
Table 5. 2022 Core Technology Discovery Project (citizen-led living lab type) evaluation indicators.
Div. Item Score Sub-items
Written assessment
(60%)		 Necessity and feasibility of the project 20 - Necessity and goals of the project
- Validity of target sites’ status and need analysis and the solutions
- Local government cooperation (support) to advance and demonstrate innovative technologies
Excellence and innovativeness of the project 20 - Excellence of innovative technology (solution)
- Demonstration and advancement plan of innovative technology (solution)
- Specificity and achievability of quantitative KPIs for advancement (problem solving)
- Economic and social effects of the innovative technology (solution)
Living lab operation plan 35 - Specificity and excellence of living lab organisational structure and collaborative governance
- Living lab operational expertise and capabilities
- Appropriateness of the living lab operation process
- Excellence of business management plan to prevent delays in project implementation
Budget investment and utilisation 15 - Company (university) investment plan
- Appropriateness of budget utilisation (living lab operation, general expenses for innovation technology advancement, prototype production budget, demonstration budget, etc.)
Sustainability 10 Specificity of post-project operation management plan
(operation management plan for adopted solutions, cooperation plan for local governments, etc.)
- Possibility of urban spread and expected effects
Presentation
(40%)		 Commitment to the project implementation 50 - Excellence and specificity of cooperation between companies/universities and local governments and project plan
- Feasibility of financing and investment plans of companies/universities, etc. and overall business budget operation when promoting sustainable projects
- Domestic and international spread plans such as commercialisation of innovative technologies (solutions)
- Sustainable smart city promotion plan (e.g., introduction of business model)
- Excellence of business management plan to prevent delays in project implementation
Expected effects and sustainability of the project
 50 - Expected outcomes (solving local problems, spreading to other local governments, etc.)
- Appropriateness and feasibility of quantitative KPIs
- Excellence in post-project operation management plan
- How to secure sustainability
Finally, the Smart Campus Challenge Project was conducted to cultivate innovation, creativity, and problem-solving capabilities across the academy by providing participation opportunities for university students in smart cities and creating a boom. An additional function of the project was to discover and demonstrate innovative solutions through students’ creative ideas and link them to preliminary start-up commercialisation. Accordingly, the Smart Campus Challenge Project differs in many evaluation items from the Smart City-Smart Town Challenges, which aim to solve the current problems in cities and regions [28].
Table 6. 2021 Smart Campus Challenge evaluation indicators.
Table 6. 2021 Smart Campus Challenge evaluation indicators.
Div. Item Score Sub-items
Prejudging
(100)		 Completeness of ideas 30 - Logics and validity of the conceptualisation of an idea and the development of a scientific theory, etc.
Creativity and challenge 25 - Originality of a new method or design that is outside the existing method
Differentiation 25 - Level of innovation or differentiation from existing or current practices
Motive of suggestion 20 - Background and purpose of idea proposal
- Relevance (fit) to smart cities and feasibility of solving urban problems
Additional points +4 - Non-metropolitan (up to 2 points)
- Smart technology company (1 point)
- Budding company (1 point)
Total 104

2.3. Smart City Policies by Country

Recently, the United Nations Framework Convention on Climate Change (UNFCCC) and the Paris Convention on Climate Change have been adopted to reduce greenhouse gas emissions, including carbon dioxide, and countries worldwide are striving to implement eco-friendly, high-efficiency energy systems. Smart-city-related policies are continuously being promoted to address this issue. Central governments actively promote technological development plans related to smart city construction [29].
The Smart City Challenge and Global City Team Challenge (GCTC) project is a representative smart city discovery demonstration project in the United States. The Smart City Challenge begins with discussions on smart solutions to solve urban problems with 15 government departments centred around the US Department of Transportation and promotes support projects by way of competition among the US cities. The program aims to build an innovation ecosystem network in the United States and worldwide. To this end, the US fosters start-ups that aim to expand to global companies and is expanding into a private-centred smart city industry that supports venture capital [30].
Singapore, an entire landmass occupied by urban spaces, has been promoting various policies for systematic urban management. In this context, a Smart Nation Plan has been established to respond to urban problems using digital technology. Singapore has promoted strong smart city policies for the systematic management of the city, including restricting vehicle ownership and securing green spaces in parks, to solve various city problems, such as low birth rates and aging populations, high population density, road congestion, and housing problems [31].
Japan’s population is expected to decrease by 20% from its current level. Despite the regional revitalisation that began in 2060 to maintain a population of 100 million, it is expected that it will be difficult to continue supplying essential city services in the future. Thus, a new smart city concept is needed to improve local quality of life using advanced technologies to realise a future society with resident consensus [32].
The Netherlands operates the Amsterdam Smart City (ASC), a public-private partnership (SPC) corporation, to solve urban problems by utilising the collective intelligence of various participants, including the government, businesses, academia, and citizens of Amsterdam. The ASC platform operates projects in six fields, with 2,000 employees, 90 partners, and over 100 projects in progress (as of 2020), including urban design, Artificial Intelligence (AI), and the Internet of things (IoT; establishing goals through smart convergence technologies such as the IoT), and is converted to a service model that provides Connectivity Digital Platform services using data.
China’s smart city policy began to be promoted in earnest in 2012, and the ‘New Urbanisation Plan’ was announced in 2014. A central government-centred policy is being promoted. Through the 13th Five-Year Plan of 2015, nationwide smart city construction was promoted until 2020, and as of 2020, the number of smart city pilot cities in China is estimated to be approximately 900. The direction of China’s smart city policy presented in the ‘New Urbanisation Plan’ is the six detailed directions of Δinformatisation of information network, Δinformatisation of planning management, Δintelligence of infrastructure, Δconvenience of public service, Δmodernisation of industrial development, and Δrefinement of social management. This plan is primarily related to information and communication technology [33].
Spain’s smart city national plan aims to support relevant industries, as well as local governments during the transition process, and promote reuse, standards, and interoperability. For the smart cityisation of local governments, support for ICT efficiency verification projects and the development of the ICT sector, standardisation, governance, and industry models are built, interoperability with the International Telecommunication Union (ITU), open data, tourist destination analysis, and rural environment citizen-centred services based on digital technologies commonly found in Spanish smart cities, such as developing and applying standardisation models with topics such as development and establishing a governance model involving all stakeholders, both public and private, through governance [34].

2.4. Research Methodology

2.4.1. Analytic Hierarchy Process

In this study, we used the AHP technique to determine the relative priority among elements constituting multiple hierarchies. AHP is a hierarchical decision-making technique that has been proven reliable [35]. The extent to which a measure fits its purpose is important because decision-making evaluates something intangible that needs to be traded off. An element for a given attribute which dominates another element is determined using a measure of absolute judgment. These judgments may be inconsistent, and AHP measures discrepancies to improve judgments where possible and obtain better consistency [36].
The basic problem of decision making involves choosing the best alternative with contradictory standards and incomplete information. The AHP provides a comprehensive framework to address these issues, allowing us to simultaneously address rational, intuitive, and irrational decisions when making multi-criteria and multi-agent decisions [37]. AHP can overcome the limitations of existing statistical analysis methods and simultaneously determine the relative importance of several criteria using a systematic and scientific method [38].
Vargas [39] proposed that the theoretical basis for the application of AHP is revealed by four axioms. First, reciprocal refers to reciprocity; the decision-maker must be able to pair and compare two factors within the same hierarchy. Furthermore, the strength of the preference must be expressible and satisfy reciprocal conditions. Second, the elements of one layer must depend on those of the upper layer. However, it is not necessary to ensure the independence between the elements in the lower hierarchy for all elements in the upper hierarchy. Third, expectations assume that the hierarchy includes matters related to decision-making. Fourth, through homogeneity, importance is expressed using a set scale within a limited range. These four axioms are necessary conditions for inducing validity in situations where the AHP is maintained or theoretically used in practice.

2.4.2. Efficiency and Productivity Measures

The Data Envelopment Analysis (DEA) technique is a nonprofit decision-making unit that performs the same or a similar function by utilising many inputs and outputs based on a system model. It is an efficiency measurement method that uses a non-parametric method to measure the relative managerial efficiency in the management operations of decision-making units (DMUs) [40].
This model has consistently been used as the most appropriate method for evaluating inefficiency in the public sector, where market prices are not formed [41]. The DEA model fixes one of the inputs and outputs and is divided into input-oriented and output-oriented models according to how inefficient parts are found for the remaining elements. Furthermore, depending on whether the effect of scale is considered when measuring efficiency, it can be divided into Charnes, Cooper and Rhodes (CCR), and Banker, Charnes and Cooper (BCC) models. The CCR model assumes constant returns to scale (CRS) for the production set, and the BCC model assumes variable returns to scale (VRS).

2.5. Literature Review

In this section, we examine the metrics related to smart cities used to derive important variables by considering the characteristics of this study. Regarding the AHP analysis, as well as efficiency and productivity analysis, a detailed review of previous studies is necessary because the selection of variables closely affects the reliability of the analysis results, including solidity and validity [42]. First, a study derived 54 indicators to constantly monitor the degree of smart city construction in global cities worldwide based on the EU’s six core factors: environment, mobility, economy, people, government, and living [43]. We examined this in detail to measure efficiency and productivity, because we recognised the need for companies to monitor and measure efficiency and productivity. Another analysis study based on 5 fields, 21 subfields, and 80 performance indicators to evaluate the governance, innovation, sustainability, connectivity, and cohesion of smart cities ensured diversity in the evaluation system [44]. Another study constructed non-technical categories of evaluation indicators, not only for construction purposes, but also to reduce operating costs and secure sustainability in the long run by building a model related to the maturity of smart cities. This implies that sustainability must be considered as well [45]. Some metrics are also organised according to the significance of performance outcomes. Since this is linked to performance indicators, this study identifies the key success factors that make projects work while adhering to the original plan [46]. Some studies derived indicators from urban perspectives. In one study, smart city indices were derived from environmental, economic, and social perspectives by various research institutes, which means that evaluation indicators can be used to select projects as well as to spread them across the city [47]. As smart cities are a convergence of digitalisation and technology, there is also an industry measurement scale, based on 10 areas and 40 indicators, to foster support for digital start-ups working on them [48]. Finally, another study ranked the top 20 global smart cities into four categories based on the performance of each city’s current operations and projects: mobility, health, safety, and productivity [49]. This indicates that smart cities are not only active in South Korea, but are also part of the global trend.
Furthermore, studies have been conducted on living labs and governance in which real residents participate to ensure sustainability. To solve urban problems using advanced technology, the problems experienced by residents in real life should be identified and solved. In this respect, to successfully promote smart city policies, it is necessary to establish a living lab and governance where citizen participation is essential [50]. Living labs support innovation by promoting existing policies, overcoming the limitations of government-led top-down promotion, and emphasising the reflection of local and citizen perspectives and public-private partnerships. As the application of living labs is expanding as a technology commercialisation and testbed for smart city experimentation and implementation, it is important to share the experience of living labs and build a network that can be applied as a new innovative model of the existing national space policy. A living lab is defined as a user-led innovation model that generally undergoes the stages of problem discovery, business planning, prototype design, product and service development, and demonstration. In this process, it develops into a win-win ecosystem in which various stakeholders exchange value at a reasonable level, premised on the continuity of service and securing a network of participants [51]. In particular, as citizens—the end users of the service—participate in the entire process of the project, continuous use and expansion of the service will be considered from the beginning of the project, which is consistent with the goal of creating a sustainable smart city. Overall, although the essential concepts of living labs and smart cities differ from the value chain viewpoint of means and ends, they complement each other in that they pursue the common goal of realising a sustainable city; these two complex application cases are expected to increase further in value. As discussed, studies on smart cities are extensive and diverse in scope and fields. However, most of these studies focused on the measurement of one or two metrics. Smart cities should be approached from the perspectives of convergence, service operations, software, IT evaluation, operational types, and sustainability. For these projects to be sustainable, it is necessary to develop more innovative and active projects, including private participation, rather than building infrastructure in the public sector, based on an understanding of the operation and management status of companies. This study is different from previous studies as it derives priorities by weighting convergence and convergence aspects to standardise the evaluation system, and it presents results to secure sustainability by analysing the efficiency and productivity of Korean smart city convergence alliance companies.

3. AHP analysis results

3.1. Organisation of Hierarchy Diagram

The hierarchy diagram was structured based on the evaluation indicators mentioned in the theoretical consideration section, while considering the sub-items of the metrics of previous studies. In particular, the metrics were reclassified to account for the scoring and items used to derive the tiers. Tier 1 is shown in Table 7.
Table 7. Evaluation system (Tier 1).
Table 7. Evaluation system (Tier 1).
Smart Challenge Project Evaluation items Score Reclassification(Tier 1)
City Town Company-led Citizen-led living lab
● ● ● ● - Necessity and feasibility of the project 20 Technical expertise
● - Excellence and innovativeness of the innovative technology 30
● - Excellence and innovativeness of the project 20
● - Living lab operation plan 35
● - Excellence of the project plan 30 Specificity of planning
● -Adequacy of the project implementation plan 30
● -Adequacy of the project promotion plan 30
● -Adequacy of budget utilisation plan 20
● -Investment and utilisation of budget 15
● - Sustainability of the project 20 Sustainability
● -Cooperation with citizens and stakeholders 10
● -Cooperation among participating organisations 20
● -Specificity and excellence of the main project implementation plan 20
● ● ● ● -Expected effects and sustainability of the project 30–50
● ● ● ● - Willingness and feasibility of local governments 35–50 Scalability
● ● ● ● - Willingness of private participants to commercialize 35–50
Based on the Tier 1 results, we conducted five rounds of FGIs with a group of experts to reflect these results and assigned standardised weights to reflect the situation in South Korea, thus finalising the hierarchy diagram including Tiers 2 and 3 (Table 8).
Table 8. Final hierarchy diagram.
Table 8. Final hierarchy diagram.
Goal Comparison of importance
(Tier 1)		 Comparison of importance
(Tier 2)		 Comparison of importance
(Tier 3)
Smart city evaluation indicators Technical expertise Technology acceptability Possession of specialized ICT technicians, establishment of a dedicated smart city department, existence of interconnected services between departments, and use of big data in policy proposals by field.
Administrative accessibility Organize smart city living labs and governance, facilitate citizen policy input, and computerize administrative services
Plan specificity Economic infrastructure Build a virtuous employment ecosystem, open consumption behaviour, local productivity, and industry spillovers
Urban infrastructure Validity of building intelligent facilities, enterprise management system (EMS), sustainability of transportation facilities
Educational infrastructure Introduce specialised ICT training and E-learning, provide intelligent educational facilities
Sustainability Social infrastructure Operate integrated operational centre and sustainable social council
Living infrastructure Smart city planning, smart healthcare and safety management
Environmental infrastructure Sustainability of environmental facilities, smart environmental management, energy management
Scalability Fostering specialised human resources Technology convergence workforce education, regional networked innovative workforce
Creation of industrial ecosystem Strengthen public-private/private-private partnerships capabilities, support for overseas export of innovative products

3.2. Analysis Result

The questionnaire was distributed to 100 experts working in smart-city-related fields. Of the 100 returned responses, 83 valid samples were used, after excluding inconsistent and incomplete responses (17). Respondents included 23 university professors, 20 civil servants, 17 general employees, 12 researchers, and 11 public companies. The AHP results, which maintained the independence of each tier are shown in Table 9.
Table 9. Final prioritisation results.
Table 9. Final prioritisation results.
Evaluation Goal
(Tier 1)		 Evaluation Criteria [Main]
(Tier 2)		 Evaluation Criteria [Sub1]
(Tier 3)		 Evaluation Criteria [Sub2]
(Tier 4)		 Final Weight Tier
Ranking
Improvement of
smart city project planning evaluation system		 Technical expertise
(L: 0.16)		 Technical acceptability
(L: 0.66)		 Possession of ICT specialized technical personnel (L: 0.24) 0.0253 15
Establishment of a dedicated smart city department (L: 0.38) 0.0401 8
Existence of interconnected services between departments (L: 0.22) 0.0232 18
Utilisation of big data for policy proposals by field (L: 0.16) 0.0169 25
Administrative accessibility
(L: 0.34)		 Smart city living lab and governance organization (L: 0.48) 0.0261 12
Ease of reflecting citizen policy opinions (L: 0.34) 0.0185 24
Computerisation of administrative services (L: 0.18) 0.0098 27
Specificity of plan
(L: 0.30)		 Economic infrastructures
(L: 0.29)		 Building a virtuous employment ecosystem (L: 0.30) 0.0261 12
Open consumption behaviour (L: 0.18) 0.0157 26
Local productivity (L: 0.23) 0.0200 23
Industrial spillovers (L: 0.29) 0.0252 16
Urban infrastructures
(L: 0.57)		 Feasibility of building intelligent facilities (L: 0.40) 0.0684 4
Enterprise management system (EMS) (L: 0.40) 0.0684 4
Sustainability of transportation facilities (L: 0.20) 0.0342 10
Educational infrastructures
(L: 0.14)		 Introduce ICT professional training and E-learning (L: 0.51) 0.0214 21
Provide intelligent educational facilities (L: 0.49) 0.0206 22
Sustainability
(L: 0.33)		 Social infrastructures
(L: 0.41)		 Operate an integrated operations centre (L: 0.59) 0.0798 3
Operate sustainable social councils (L: 0.41) 0.0555 6
Living infrastructures
(L: 0.38)		 Establish smart city plans (L: 0.79) 0.0991 1
Smart medical and safety management (L: 0.21) 0.0263 11
Environmental infrastructures
(L: 0.21)		 Sustainability of environmental facilities (L: 0.36) 0.0249 17
Smart environmental management (L: 0.33) 0.0229 19
Energy management (L: 0.31) 0.0215 20
Scalability
(L: 0.21)		 Nurture professional workforce
(L: 0.31)		 Technology convergence workforce education (L: 0.61) 0.0397 9
Regional networked innovative workforce (L: 0.39) 0.0254 14
Creation of industrial ecosystems (L: 0.69) Strengthening public-private/private-private partnerships (L: 0.68) 0.0985 2
Support for overseas export of innovative products (L: 0.32) 0.0464 7
From the AHP, smart city planning (0.9991) ranked first, followed by public-private/private-private partnership capacity building (0.0985), integrated operation centres (0.0798), feasibility of building intelligent facilities (0.0684), and EMS (0.0684). This analysis reflects the recent trends of all local governments in Korea regarding the establishment of smart city plans. Therefore, it is desirable to promote smart city projects with a long-term sustainability plan, rather than as a one-off event. Public-private/private-private partnership capabilities should be strengthened to ensure project scalability. Ultimately, securing profitability and stability seems impossible when the private sector participates in and leads projects. The feasibility of building intelligent facilities also seems to reflect the expert opinion that these facilities are required to address sustainability issues. Ultimately, a smart city project requires a substantial investment, and it is important to ensure cost-effectiveness. It appears that feasibility should be secured in advance, and there may be frequent failures and significant maintenance costs owing to the nature of electronics and devices. Finally, EMS is considered highly important because of the characteristics of a smart city project, which requires integration and effective control of various subjects and departments, and smooth analysis and utilisation of the database.

4. Efficiency and Productivity Analysis Results

In addition to the results of the previous analysis, we conducted an efficiency and productivity analysis on 38 SMEs that are the most sensitive to business performance among the currently operating smart city convergence alliance participants, to use as the basis for an evaluation system. The analysis period was limited to three years, from 2019 to 2021, and an analysis model was established to identify trend changes over the three years. The variables used in the analysis were capital (KRW million), number of employees (people), assets (KRW million), sales (KRW million), operating income (KRW million), and net income (KRW million). Regarding the variables, the items generally used to evaluate the management of companies were prioritised as the scope that can measure the actual management status. Although additional variables can be reflected, the analysis of efficiency and productivity can be unreliable when the number of variables is more than twice that of the DMUs. Therefore, they were not included to ensure the validity and reliability of the analytical model. The basic statistics are shown in Table 10.
Table 10. Basic statistics.
Table 10. Basic statistics.
Year Category Minimum Maximum Mean SD
2019 Capital (KRW million) 567 148,768 11,309.50 24,912.63
Number of employees (people) 1 268 54.95 71.50
Assets (KRW million) 832 235,774 18,564.68 40,117.95
Sales (KRW million) 122 94,172 12,770.98 18,872.19
Operating income (KRW million) -6,521 5,985 78.50 1,759.19
Net income (KRW million) -29,524 5,758 -1,000.20 5,231.20
2020 Capital (KRW million) 637 173,417 11,933.95 28,302.18
Number of employees (people) 1 285 62.60 75.56
Assets (KRW million) 951 251,398 20,486.45 41,861.79
Sales (KRW million) 77 91,237 15,187.32 21,907.16
Operating income (KRW million) -6499 6,576 446.50 2,150.39
Net income (KRW million) -17139 7,256 55.90 3,405.35
2021 Capital (KRW million) 699 200,921 15,099.75 33,297.94
Number of employees (people) 6 375 76.50 86.58
Assets (KRW million) 1,054 260,651 23,762.50 45,064.57
Sales (KRW million) 63 82,219 14,994.60 20,483.75
Operating income (KRW million) -7,488 25,486 968.87 4,582.89
Net income (KRW million) -9,418 19,066 810.40 4,029.82
Particular attention should be paid to operating costs and net income. The maximum annual value follows an upward trend. In particular, the figure will be approximately three times higher in 2021 than in the previous year. This implies that smart city projects will enter a steady state with increasing income from 2021 onwards. However, they are still in the red despite the decrease in the minimum value; therefore, it is urgent to address this problem through technological innovation and policies that temporarily relieve regulations to demonstrate smart city projects, among others.
Table 11. CCR analysis results.
Table 11. CCR analysis results.
DMU 2019 2020 2021
1 0.4782 0.6321 0.6951
2 1 1 1
3 1 1 1
4 1 1 1
5 0.6852 1 0.9785
6 1 1 0.9213
7 1 1 0.9921
8 1 1 1
9 0.8212 0.8631 0.8323
10 0.5733 0.5752 0.6845
11 0.632 0.4302 1
12 0.7732 0.9561 0.9429
13 1 0.9212 0.9124
14 0.8633 0.6752 0.3829
15 0.4469 0.4932 0.2764
16 0.8424 1 1
17 0.8358 0.6621 0.6348
18 0.8902 0.9631 1
19 0.6128 1 1
20 0.6332 0.4921 0.9212
21 0.9562 0.9102 0.9212
22 0.7632 0.7921 1
23 0.8236 0.6982 0.7823
24 0.9563 0.6933 0.6952
25 0.921 0.9215 1
26 0.3452 0.5231 0.6218
27 0.6213 0.5212 0.1625
28 0.5852 0.9612 0.5289
29 0.5775 0.6907 0.6619
30 1 0.584 0.7113
31 1 1 0.9932
32 0.5992 0.5551 0.6212
33 0.6132 0.4822 0.4232
34 0.6333 0.5212 0.3921
35 1 0.9212 0.8218
36 0.5236 0.4921 0.4922
37 0.792 1 1
38 0.5632 0.6341 0.7521
Average 0.7727 0.7780 0.7830
The CCR analysis results show a slight increase in efficiency from an average of 0.7727 in 2019 to 0.7830 in 2021. For each unit, DMUs 2, 3, 4, 6, 7, 8, 16, 19, 31, and 37 had a value of 1 or close to 1, indicating that they were operationally efficient. In the future, it will be necessary to select and share best practices by monitoring and consulting with these companies to identify policy factors that can help them improve their operational efficiency.
Table 12 shows the results of the BCC analysis, categorised into technical efficiency (TE), pure technical efficiency (PTE), and scale efficiency (SE). TE was 0.8102, PTE and SE were 0.0973 and 0.8642, respectively; when the value is close to 1, it is considered efficient. Under the CRS and VRS assumptions, smart city convergence alliance SMEs were found to maximise their efficiency by reducing their inputs by 81% and 86%, respectively. As all values are on the upward curve from year to year, operational efficiency is expected to improve gradually.
Table 12. BCC analysis results.
Table 12. BCC analysis results.
Category 2019 2020 2021 Average
TE 0.7843 0.7932 0.853 0.8102
PTE 0.8671 0.8522 0.8732 0.8642
SE 0.8782 0.9135 0.9301 0.9073
N 38 38 38 38
TE: technical efficiency; PTE: pure technical efficiency; SE: scale efficiency; N:.
Table 13. Analysis of return to scale.
Table 13. Analysis of return to scale.
Category 2019 2020 2021 Sum
CRS 9 11 11 31(27.19%)
DRS 11 10 12 33(28.94%)
IRS 18 17 15 50(43.87%)
N 38 38 38 114(100%)
CRS: constant returns to scale; DRS: ;IRS: ; N:.
Finally, the three-year time-series average productivity index are shown in Table 14.
Table 14. Malmquist productivity index change rate.
Table 14. Malmquist productivity index change rate.
Time-series TECI TCI PECI SECI MPI
T2 0.9893 1.0555 0.9526 1.0386 1.0442
T3 0.9538 0.9482 0.9339 1.0213 0.9044
Geometric mean 0.9714 1.0004 0.9432 1.0299 0.9718
TECI: ;TCI: ; PECI: ; SECI: ; MPI: municipal performative index; T2: ; T3:
Finally, for productivity analysis, the degree of change between the two time points was measured. If the municipal performative index (MPI) value of the time series at T2 and T3 is greater than 1, this indicates a mutual increase in productivity at two separate points in time, t and t+1, from 2019 to 2020 and 2020 to 2021, respectively. In other words, when the value is greater than 1, productivity increases; when it is 1, there is no change between the time points; and when it is less than 1, productivity decreases. After analysis, T2 showed an increase in productivity, whereas T3 showed a decrease. Taken together, these efficiency results suggest that the problem is more operational than technical. Some companies have technology that cannot be demonstrated because of legal and institutional barriers, and recently, external factors such as inflation, high interest rates, and labour shortages also seem to be involved.

5. Discussion

5.1. Summary and Significance of the Study

This study identifies, analyses, and prioritises the significant factors for standardising the Korean smart city project evaluation system. In addition, we analyse the operational performance of companies currently providing smart city services in terms of efficiency and productivity to consider both policy and practical aspects. This is a timely study, as a smart city project is not a one-time project but directly affects the quality of life of local residents. Furthermore, given the characteristics of smart city projects that require substantial resources and that are implemented by several departments simultaneously, studying the standardisation of the evaluation system is necessary. Most countries, including developing countries, are carrying out smart city initiatives. The development of global standards should also be discussed. In this context, we examined the evaluation factors and elements derived from previous studies and smart city evaluations conducted in South Korea, based on which a hierarchical diagram was constructed.
Twenty-seven major factors were derived and confirmed through five rounds of FGIs, and with experts identifying the priorities. In addition, we conducted an efficiency and productivity analysis of 38 SMEs involved in smart city convergence alliances and used them as the basis for an evaluation system. The analysis is not limited to a short-term perspective but presents both individual efficiency and overall productivity analysis results over time. This study is expected to serve as the basis for building a standardised evaluation system that includes both policy and practical implications for companies. The AHP showed that smart city planning (0.9991) ranked first, followed by public-private/private-private partnerships capacity building (0.0985), integrated operation centres (0.0798), feasibility of building intelligent facilities (0.0684), and EMS (0.0684). This result reflects that the eligibility requirement for local governments to apply for the Korean Smart City Project contests was specified as the establishment of a smart city plan rather than a general city plan. In some cases, only additional points were given; however, recently, all local governments have been establishing smart city plans, and this analysis seems to reflect this trend. Therefore, it is desirable to promote smart city projects with a long-term sustainable plan rather than as a one-off event, as indicated by the analysis results. Public-private/private-private partnership capabilities should be strengthened to ensure project scalability. Until now, the public sector has led smart city projects to support the basic infrastructure. However, this has had its side effects, such as inconsistencies in projects, difficulty with the introduction of innovative technologies, and one-off election-promoting projects. Ultimately, projects led by the private sector seem more stable. The feasibility of building intelligent facilities also reflects the expert opinion that it is necessary to address the issue of sustainability. Ultimately, a smart city project requires huge investments, and it is important to ensure cost-effectiveness. Feasibility should be secured in advance, as there may be frequent failures and significant maintenance costs owing to the nature of electronics and devices. Finally, an EMS is deemed highly important because of the characteristics of a smart city project, which requires integration and effective control of various subjects and departments, and smooth analysis and utilisation of the database. In addition to the previous analysis results, we conducted an efficiency and productivity analysis on 38 SMEs that are the most sensitive to business performance among the currently operating smart city convergence alliance participants and used them as the basis for an evaluation system. The analysis period was limited to three years from 2019 to 2021, and an analysis model was established to identify trend changes over the three years. According to the analysis results, under the CRS and VRS assumptions, smart city convergence alliance SMEs can maximise their efficiency by reducing their input by 81% and 86%, respectively. However, as all values are on the upward curve from year to year; the operational efficiency is expected to gradually improve.
The implications and policy recommendations derived from these results are as follows. First, to improve the smart city project plan evaluation system, project evaluation, and to revitalise the smart city industry, it is necessary to supplement the evaluation index by considering the relative importance of the indices to ensure that the sustainability of projects can be evaluated. Furthermore, because the purpose of the Smart Challenge project is to utilise corporate solutions based on the ideas of universities and citizens regarding urban problems, evaluation indicators for companies must be considered for the smooth promotion of the project.
Second, based on the smart city planning of local governments, standard guidelines were prepared to promote business expertise, business plan specificity, and business scalability, and among companies with smart city technologies, to expand specialised technologies for SMEs. To secure a new market, institutional support, such as financial support for commercialisation from central and local governments, tax benefits, incentives, and living labs involving residents, should be strengthened along with expert training.
Third, the results indicate that smart city-related companies in Korea improve productivity by increasing efficiency through operational rather than technological improvements. Therefore, innovative technologies such as digital twins, platforms, IoT, AI, big data, and Geographic Information Systems/ Global Positioning Systems are advanced, and joint businesses are strengthened, while the legal and institutional aspects of actual operations are flexible as the basis for securing profitability. Policy alternatives are required, such as the preparation and activation of policies that temporarily relieve regulations to demonstrate smart city projects.

5.2. Research Limitations and Future Tasks

This study had some limitations. Numerous smart city projects should not be viewed simply, but from multiple perspectives; therefore, their purpose and direction are not always clearly aligned. In addition, although we set up a research model in several ways and presented each analysis result, we could not score or standardise them. Furthermore, AHP, efficiency, and productivity analyses are based on qualitative and quantitative data, respectively; therefore, applying the same point in time to the same variable is problematic. In particular, in efficiency and productivity analyses, the failure to classify the types of smart city convergence alliance companies should be addressed. Future researchers could benefit from providing a three-dimensional perspective by addressing the limitations of this study.

References

  1. Myeong, S.; Jung, Y.; Lee, E. A study on determinant factors in smart city development: An analytic hierarchy process analysis. Sustainability 2018, 10, 2606. [Google Scholar] [CrossRef]
  2. Telecommunications Technology Association. Key convergence cases of the 4th industrial revolution smart city concept and standardization status, standardization issue 2018-1, 2018. Available online: URL (accessed on Day Month Year).
  3. Seoul City Council. A study on analyzing and solving social problems in the digital city of Seoul, 2018. Available online: URL (Accessed on Day Month Year).
  4. Peter Hansen, L. Repercussions of pandemics on markets and policy. Rev. Asset Pricing Stud. 2020, 10, 569–573. [Google Scholar] [CrossRef]
  5. Belanche, D.; Casaló, L.V.; Orús, C. City attachment and use of urban services: Benefits for smart cities. Cities 2016, 50, 75–81. [Google Scholar] [CrossRef]
  6. Hwang, J.; Jang, J. Prospects for smart city development and Korea’s competitiveness. IT & Future Strategy. 2016 No. 6; National Information Society: Daegu Agency.
  7. Kim, Y.; Koo, J. Study on the developing of evaluation indicators for smart city from the perspective of digital social innovation. J. Korea Contents Assoc. 2019, 19, 512–521. [Google Scholar]
  8. Han, S.; Shin, Y.; Yu, I.; Lee, J. A study on the Korea smart city certification index and demonstration authentication. J. Korea Acad.-Ind. Coop. Soc. 2018, 19, 688–698. [Google Scholar]
  9. Ahn, Y.; Lee, S.; Yu, M.; Jeong, G.; Yeom, I.; Ji, N.; Kim, A. Establishment of a smart city model based on citizen participation; Daejeon Sejong Institute: Daejeon, 2018. [Google Scholar]
  10. Jin, S. Life satisfaction depending on digital utilization divide within people with disabilities. Informatization Policy 2019, 26, 69–89. [Google Scholar]
  11. Ministry of Land, Infrastructure and Transport, 2022. Smart city industry promotion strategy research final report. Available online: URL (accessed on Day Month Year).
  12. Dutton, W.; Blumler, J.; Kraemer, K. Wired cities: Shaping the future of communications; G.K. Hall: New York, USA, 1987. [Google Scholar]
  13. Ishida, T. Understanding digital cities. In Digital cities: Experiences, technologies and future perspectives lecture notes in computer science; Ishida, T., Isbister, K., Eds.; 1765. Springer-Verlag: New York, USA, 2000. [Google Scholar]
  14. Aurigi, A. Competing urban visions and the shaping of the digital city. Know. Techn. Pol. 2005, 18, 12–26. [Google Scholar] [CrossRef]
  15. Mora, L.; Bolici, R.; Deakin, M. The first two decades of Smart-city research: A bibliometric analysis. J. Urban Technol. 2017, 24, 3–27. [Google Scholar] [CrossRef]
  16. Yigitcanlar, T.; Kamruzzaman, M.; Foth, M.; Sabatini-Marques, J.; da Costa, E.; Ioppolo, G. Can cities become smart without being sustainable? A systematic review of the literature. Sustain. Cities Soc. 2019, 45, 348–365. [Google Scholar] [CrossRef]
  17. OECD. Smart cities and inclusive growth; OECD: Paris, 2020; Available online: URL (Accessed on Day Month Year). [Google Scholar]
  18. ITU-T, 2014. Smart sustainable cities: An analysis of definitions [ITU-T Focus Group on Smart Sustainable Cities Technical Report]. Available online: URL (Accessed on Day Month Year).
  19. Cohen, B. The 3 generations of smart cities: Inside the development of the technology driven city, 2015. (accessed on Day Month Year). Available online: https://www.fastcompany.com/3047795/the-3-generations –of-smart-cities.
  20. Nam, T.; Pardo, T.A. Conceptualizing smart city with dimensions of technology, people, and institutions, The 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times, College Park Maryland, USA, 2011.
  21. Kang, D.; Choi, C.; Lee, J.; Lee, J.S.; Park, S. Employment impact assessment study for the global smart city demonstration complex project; Korea Labor Institute: Sejong, 2016. [Google Scholar]
  22. Karvonen, A.; Cugurullo, F.; Caprotti, F. Inside smart cities: Place, politics and urban innovation; Routledge: London, 2019. [Google Scholar]
  23. Eip, S.C.C., 2018. European innovation partnership on smart cities and communities, General Assembly 2018 Summary Report, Sofia, Bulgaria. Available online: URL (accessed on Day Month Year).
  24. Jung, S. Policy trends for smart city promotion in Europe. Technol. Trends Wkly 2019, 1921, 2–13. [Google Scholar]
  25. Ministry of Land, Infrastructure and Transport 2021. Announcement of the 2021 smart city challenge project competition. Available online: URL (accessed on Day Month Year).
  26. Ministry of Land, Infrastructure and Transport 2021. Announcement of the 2021 smart town challenge project contest. Available online: URL (accessed on Day Month Year).
  27. Ministry of Land, Infrastructure and Transport 2022. Announcement of competition for smart city innovative technology discovery project in 2022. Available online: URL (accessed on Day Month Year).
  28. Ministry of Land, Infrastructure and Transport 2022. Announcement of competition for regional base and small city smart city creation project in 2022. Available online: URL (accessed on Day Month Year).
  29. Kim, K.; Kim, G. Status and prospect of smart city in the fourth Industrial Revolution. J. Korean Converg. Soc. 2018, 9, 191–197. [Google Scholar]
  30. Lee, J.; Lee, M.; Lee, J.; Kim, I. Strategic response measures according to smart city types, Korea Research Institute for Human Settlement, 2018.
  31. Cho, J.; Lee, N.; S.; W. Smart city overseas case study in Singapore, smart city policy and governance research, Seoul Digital Foundation, 2021.
  32. Cho, M.; Lee, J., 2021. Contents and implications of the Japanese super city initiative to resolve resident-oriented local challenges, National land issue [Report]; Volume 42.
  33. Lee, H.; Bang, S.; Kim; Oh, H.; Wang, F. China smart city promotion status and entry strategies: Focusing on the cases of Xiong’anjin-gu and Tianjin Eco-city; Korea Institute for International Economic Policy and Korea Research Institute for Human Settlements, 2021.
  34. Korea Research Institute for Human Settlements; Smart City Association, 2022. Smart City Policy Forum.
  35. Saaty, T.L. Rank generation, preservation, and reversal in the analytic hierarchy process. Decis. Sci. 1987, 18, 157–177. [Google Scholar] [CrossRef]
  36. Saaty, T.L. Decision making with the analytic hierarchy process. Int. J. Serv. Sci. 2008, 1, 83–98. [Google Scholar] [CrossRef]
  37. Saaty, T.L. Axiomatic foundation of the analytic hierarchy process. Manage Sci 1986, 32, 841–855. [Google Scholar] [CrossRef]
  38. Woo, C.; Kim, G.; Kang, S. Comparative study of bankruptcy prediction models using LOGIT analysis and AHP analysis. Financ. Manag. Res. 1997, 14, 229–252. [Google Scholar]
  39. Vargas, L.G. An overview of the analytic hierarchy process and its applications. Eur. J. Oper. Res. 1990, 48, 2–8. [Google Scholar] [CrossRef]
  40. Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978, 2, 429–444. [Google Scholar] [CrossRef]
  41. Kim, S. Structure analysis of production efficiency of local public service supply, Korean. J. Local Auton. 2000, 12, 47–65. [Google Scholar]
  42. Shim, H.; Kim, J. Measures for the improvement of feasibility studies and investment review: Identification and verification of major project sectors considering balanced regional development. Sustainability 2020, 12, 9531. [Google Scholar] [CrossRef]
  43. Smart Cities Council. Smart cities index. Available online: URL (accessed on Day Month Year), 2011.
  44. INTELI. Towards a smart cities index: The case of Portugal. Available online: URL (accessed on Day Month Year), 2012.
  45. IDC. Smart cities and the Internet of everything: The foundation for delivering next-generation citizen services. Available online: URL (accessed on Day Month Year), 2013.
  46. ITU-T. Global smart sustainable city. Available online: URL (accessed on Day Month Year), 2016.
  47. EU city keys, 2017. Co-funded by the European Commission within the H2020 Programme. Available online: URL (accessed on Day Month Year).
  48. Nesta. What next for digital social innovation. Available online: URL (accessed on Day Month Year), 2017.
  49. Juniper research, 2017. Smart cities – What’s in it for citizens? Available online: URL (accessed on Day Month Year).
  50. Chang, H. Study on issues and perception changes in smart cities: Focusing on news, blogs, and twitter data, cadastral and territorial. Information 2019, 49, 67–82. [Google Scholar]
  51. Seong, J.; Park, I. ICT Living Lab case analysis and implications as a user-led innovation model. Sci. Technol. Research 2015, 15, 245–278. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated