Prerpints.org logo
Preprint
Article

Vulnerable Road User Safety Using Mobile Phones with Vehicle-to-VRU Communication

Altmetrics

Downloads

99

Views

33

Comments

0

A peer-reviewed article of this preprint also exists.

Submitted:

05 December 2023

Posted:

06 December 2023

You are already at the latest version

Alerts
Abstract
Pedestrians, bicyclists, scooterists are Vulnerable Road Users (VRU) in traffic accidents. The number of fatalities and injuries in traffic accidents involving vulnerable road users has been steadily increasing in the last two decades in the U.S. even though road vehicles now have perception sensors like camera to detect risk and issue collision warnings or apply emergency braking. Perception sensors like camera are highly affected by lighting and weather conditions. Camera, radar and lidar cannot detect vulnerable road users in partially occluded and occluded situations. This paper proposes the use of Vehicle-to-VRU communication to inform nearby vehicles of VRUs having trajectories with potential collision risk. An Android smartphone app with low energy Bluetooth (BLE) advertising is developed and used for this communication. The same app is also used to collect motion data of VRUs for training. VRU motion data is smoothed using a Kalman filter and an LSTM neural network is used for future motion prediction. This information is used in an algorithm comparing Time-To-collision-Zone (TTZ) for the vehicle and VRU and issue a driver warning with different severity levels. The warning severity level is based on analysis of real data from a smart intersection for close vehicle and VRU interactions. The resulting driver warning system is demonstrated using proof-of-concept experiments. The method can easily be extended to a VRU collision mitigation system.
Keywords: 
Subject: Engineering  -   Transportation Science and Technology

1. Introduction

A significant number of traffic accidents occur each year and result in a large number of injuries and deaths [1]. To reduce these incidents, researchers have been introducing new and better active and passive protection measures each year to improve passenger and other road user safety [2], [3], [4]. In traffic, pedestrians are among the most vulnerable road users (VRU) [5], [6], [7], and pedestrian safety is a priority for the U.S. Federal Highway Administration’s (FHWA) Office of Safety [8]. The City of New York, for example, is testing new technologies to improve the safety of pedestrians [9]. According to the National Highway Traffic Safety Administration (NHTSA) statistics, more than 6,000 pedestrians are killed in traffic accidents every year in the U.S. Although pedestrians are the most common VRU type, other VRUs such as bicyclists [10] and scooter users [11] are also very susceptible to high collision risk scenarios and crashes.
To improve the safety of VRUs as they interact with autonomous and non-autonomous vehicles, an important tool that can be utilized is vehicle-to-everything (V2X) communication [12], [13], [14], [15]. This can be achieved on an individual basis where each road user carries a message broadcasting device, on an infrastructure basis where an intersection is designed as a smart intersection that detects VRUs as well as vehicles and publishes information for them as if they are all connected [16], [17], or through over the air sharing of relevant Connected Vehicle (CV) data in centralized servers. Although an automated vehicle can obtain more accurate information about the VRUs using its own sensors, there are conditions where this is not reliable or possible, such as bad weather (heavy fog, rain), or a no-line-of-sight (NLOS) situation where the target of interest is blocked by an obstacle or another vehicle [18]. When the information is transferred via wireless communication, these types of situations can be remedied. Since availability of Bluetooth advertising mode and Bluetooth 5.0 technology in recent mobile phone models is getting more widespread [19], [20], this technology is also becoming a viable means of communication between VRUs and other road users [21]. Mobile phones can be carried by any VRU on hand or in pocket, and they have a wide array of sensors available for measurement for tracking or predicting the movement and behavior of the pedestrian, bicyclist or scooterist [22], [23]. Using Android Bluetooth Low Energy (BLE) application programming interface (API) [24], an Android app can be developed and used for this purpose. The different modes of communication under Bluetooth can provide different benefits when using this approach [25].
Pedestrian safety systems need knowledge of the possible future path of nearby pedestrians to determine the possibility of a near future collision risk [26], [27], [28]. Even with the correct information transfer from a pedestrian’s phone using Bluetooth or other communication means, prediction of the path of the VRU can be tricky, especially for pedestrians [29]. Different types of methods such as grid based ones [30], constant velocity models [31], neural networks [32] have been utilized in the literature to predict the pedestrian path. Generative Adversarial Networks (GAN) [33] and Long Short-Term Memory (LSTM) networks [34] have been used and observed in the literature to be providing reliable results for prediction. The memory-based feedback nature of the LSTM network helps with the motion patterns of pedestrian movements, especially at intersections. The prediction method can be combined with the data obtained from sensors or wireless communication. One problem posed here could be the low frequency update rate of 1 Hz of normal GPS sensors if the pedestrian is the source of information. However, GPS and acceleration and gyro sensors available on the phone can be utilized with a Kalman Filter [35] for higher frequency location tracking. Utilizing sensors in the pedestrian’s mobile phone is very beneficial when combined with Bluetooth advertisement capabilities. This approach helps ensure information transfer between road users and, most importantly, helps enable the implementation of safety applications that can conveniently be installed in mobile phones which can be used by both the driver of a vehicle and also by the VRU.
This paper focuses on VRU safety using V2P communication. The methods and results presented in the paper are applicable to all VRUs but the experimental results reported here are for pedestrians. The contributions of the paper are:
  • the development of an Android app for collecting mobile phone sensor data for VRUs,
  • using the collected data to predict the future motion of a VRU using Kalman filtering and LSTM based motion prediction,
  • converting this future motion and localization data of the VRU to a Personal Safety Message (PSM),
  • broadcasting the PSM using Bluetooth advertising or over-the-air using internet connectivity to nearby vehicles,
  • Further developing and using the same Android app to receive PSM data in a vehicle,
  • developing and implementing programs in a Bluetooth board to similarly broadcast and receive PSM data if needed,
  • developing and implementing a pedestrian collision warning system using the PSM data,
  • analyzing available vehicle and pedestrian interactions in a smart intersection to calibrate the pedestrian collision warning system, and
  • developing and demonstrating a full pedestrian collision avoidance system experimentally.
V2P communication is used to improve the safety of VRUs, especially pedestrians, using a driver warning system in this paper. The VRU mobile phone sensors are used to predict future motion to better assess collision risk with nearby vehicles and issue warning messages. These warnings can also be used to begin automated slowing down and braking of the vehicle. The organization of the rest of the paper is as follows. V2P app development and implementation is presented in Section 2. V2P communication using the developed app is investrigated in Section 3. The V2P communication based pedestrian safety system is developed in Section 4. Simulation and experimental results are presented and discussed in Section 5. The paper ends with conclusions and recommendations in Section 6.

2. V2P App Development and Implementation

To determine collision risk possibility and execute the safety precautions, pedestrian information needs to be obtained by the vehicle. By transferring it over a wireless link, we also ensure that even if there is NLOS and perception sensors cannot detect the pedestrian, the vehicle would know that there is a pedestrian and be aware of his/her motion state such as location, speed and heading. Therefore, the main enablers of this implementation are the capabilities and relatively wide availability of Bluetooth 5.0 in recent Android phone models. However, it should be noted that the algorithms and methods developed in this paper for vehicle and pedestrian interaction analysis, driver alert and warning, pedestrian motion and intent prediction and collision risk determination are agnostic of the communication method used.
Bluetooth communication technology was experimented to analyze practical capabilities of its implementation in smartphones. First, communication experiments were done in a Bluetooth development board: nRF52840-DK [36]. Afterwards, an Android application was coded using the BLE module in Android API 28. This software has the capability to encode, transmit, receive and decode messages via Bluetooth 5.0 through extended advertisement, depending on the selected user. For the message structure, PSM from the Society of Automotive Engineers (SAE) J2735 standard [37] was preferred to convey the pedestrian information. For the mode of Bluetooth communication, experiments were first done with extended advertisement due to increased distance and larger data carrying capacity and very good results were obtained. In order for our approach to be applicable for a large range of smartphones, however, the mode of communication was switched later to normal advertisement as it is available in more smartphones. Along with having location, speed, acceleration and heading fields for calculating the collision possibility and warning, PSM also has fields such as user type, device use state, cross request, cluster size and attachment. All of these fields are useful for future improvements of the algorithm regarding behavior prediction, pedestrian group clustering and more accurate collision prediction in cases where the VRU has attachments such as pets, carts or wheelchair. The standard is designed for this message to be published at 10 Hz which is a reasonable frequency for our calculations. Since only the driver is warned in this implementation, the pedestrian is the transmitting side, and the vehicle is the receiving side. This communication is illustrated in Figure 1. Populated fields for PSM are the location (latitude and longitude), heading, speed and acceleration as shown in Figure 2.
On the vehicle side, an Android phone can be used to obtain the pedestrian information and handle the warning calculations and show warning and recommended speed profile (if desired) to the driver. If the vehicle is autonomous, a Bluetooth receiver board such as the nRF52840-DK discussed above can be used to obtain the messages and transfer the information to computing unit(s to process for comfortable braking and stopping as well as for collision avoidance. The board has a serial interface but it was combined with a W5500 Ethernet shield [38] to transfer the information received via Bluetooth through the Ethernet connection for better results. The picture of the combined BLE receiving board can be seen in Figure 3.

3. V2P Communication Experiments

After the implementation of V2P into the smartphone, some initial testing for determining communication range was conducted. This testing involved both nRF52840-DK Bluetooth boards and Android smartphones in obstructed and non-obstructed cases along with normal and extended Bluetooth advertisement modes. The results obtained are presented in Table 1. The Bluetooth board use results in the longest communication range which increases to more than 250 m in extended advertising mode. This corresponds to the first row of the table Obstruction of the smartphone by the pedestrian body which would occur if the phone was in the rear pocket results in the lowest communication range in normal Bluetooth advertising mode when communicating with another smartphone in the vehicle. This corresponds to the last row of the table. Data transfer over WebSocket [39] functionality was also implemented in the application. However, this was changed later to internal storage during the experiments because the WebSocket connection required constant internet access. Moreover, due to GPS sensor updates being paused when the phone screen is locked in the initial direct implementation, GPS callback was tied to a background process to have updates regularly. A process flow diagram was prepared for the application for more detailed information and can be seen in Figure 4.
Along with the V2P broadcasting mobile application discussed above, a data collection application was developed and implemented in the Android phone. This application was developed separately from the previous app to support older model Android phones which do not have Bluetooth 5.0. The application running on an Android phone is displayed in Figure 5. This application receives data from several sensors built in the Android phone and records them into the local storage with timestamps as csv files. The sensors used are acceleration, gyro, step counter, GPS, orientation, proximity, and light sensor. A feature for uploading the data into a remote server was also implemented. The plotted data after initial testing of the app is shown in Figure 6.

4. V2P Communication Based Pedestrian Safety Warnings

The two applications discussed previously for V2P communication and data recording were later combined into a single application built for recent model Android phones with Bluetooth 5.0. The user interface was also improved. Then, a custom J2735 library was coded to encode and decode PSM messages. Several quality-of-life changes were implemented later on, such as manual or automatic user selection, manual or random PSM ID, easier file access for recorded data, debug mode for developers and user mode for end users. The app screen displayed when the improved application starts is shown in Figure 7. The features of the previously mentioned two applications, PSM decoding, encoding, broadcasting and data collection were combined into this final application. On top of that, the user interface was simplified and improved. Most importantly, the designed driver warning system, which will be discussed later in the paper, and icons were implemented to make this application warn the human drivers when there is a collision risk. Warning design, parameters and the user interface will be discussed in more detail later in the paper.

4.1. Safety Approach

The overall architecture of the V2P communication based pedestrian safety approach of this paper is illustrated in Figure 8 for a scenario in which the vehicle is approaching a crosswalk while a pedestrian is intending to cross it. The elements that make up this architecture are explained next. The first element in the approach, pedestrian path prediction and tracking, is utilized to predict the pedestrian’s location in the near future with LSTM neural networks, as well as to track his/her motion at higher sampling rate utilizing the Kalman filter, to make decisions about the future outcome of the safety system and calculations. These include how to warn the driver or how to adjust the speed of the vehicle if it is autonomous.
The second element in the architecture, communication or information transfer focuses on wireless communication to provide information about the pedestrian, especially in dangerous cases such as reduced visibility or the NLOS situation, to the nearby vehicle. Bluetooth and extended Bluetooth technology in mobile phones are used in this paper due to enhanced support to carry larger chunks of information over longer distances. On top of that, with the dual-frequency GPS technology recently starting to be implemented in mobile devices, localization accuracies of a few decimeters can be achieved [40]. Most importantly, mobile phones are very common and can be utilized by non-autonomous vehicles as well for driver warning. Because of this, the means of wireless communication to discuss and demonstrate this approach is chosen as Bluetooth in this paper. However, it should be noted that the VRU collision warning and avoidance braking methods of this paper do not depend on the communication method used.
The next two elements of the architecture, real-world data analysis and driver warnings, are intertwined with each other. The warning system was designed to allow non-autonomous vehicles to utilize this safety system by showing the driver a warning with varying degree of severity of the collision risk situation. To determine the severity of these warnings, real-world data analysis plays a very important role. This analysis process allowed us to capture the interaction between the vehicles and pedestrians at an intersection in order to come closer to a natural approach in slowing down and stopping that is achieved by providing the driver the correct warnings at the right times.

4.2. Pedestrian Path Tracking and Prediction

The path tracking and prediction system is designed to be a combination of Kalman filtering and neural network based path prediction. While neural network based prediction provides predicted motion information multiple seconds into the future, Kalman filtering increases the frequency of location updates that are normally at 1 Hz due to the present GPS update rate in mobile devices. The diagram of the neural network and Kalman filter path tracking and prediction algorithm is shown in Figure 9. The mobile phone GPS sensor is utilized at its maximum rate of 1 Hz to obtain location and velocity information which is then fed into the LSTM neural network to predict the future location of the pedestrian. Although a single neural network can be used to predict multiple locations on the path for the pedestrian, multiple and different neural networks which were trained separately yielded better prediction accuracy with smaller network size. Therefore, the algorithm uses multiple small neural networks to predict future location up to a few seconds ahead. Here the tracking is for 1 sec ahead and 2 sec ahead. Naturally, the farther the algorithm tries to predict, the less accurate it gets. Along with the neural network prediction, the Kalman filter uses GPS information at 1 Hz as well as other higher update rate sensors such as orientation and acceleration to track the pedestrian location at higher sampling than 1 Hz.
Regarding the type of neural networks, the LSTM type recurrent neural network was selected to be used in order to inherit the memory nature of the network since pedestrian movement patterns highly benefit from both short and also long term memory, especially at an intersection. LSTM was also tested in other research in the field [34] and shown to be performing better for pedestrian path prediction as compared to other options. These networks were trained with both real-world data collected from mobile phones and also synthetically generated pedestrian data from the Vissim environment. Some of these samples are presented as an example in Figure 10. The trained neural network can be transferred into mobile devices using TensorFlow Lite library for runtime operation [41].

4.3. Pedestrian Behavior Prediction

Android phones have numerous sensors that provide high frequency data. Some of these sensors were utilized to predict pedestrian behavior. An LSTM based neural network structure was created with these sensor readings being used as multiple inputs. The first structure experimented with had a single output with values between 0 and 1. Different intervals within this range were trained as different behaviors. This structure with the inputs and outputs is illustrated in Figure 11.
To train this neural network of Figure 11, real-world data was collected with the data collection application developed and explained earlier. Afterwards, the data was manually labeled utilizing a custom data labeling tool developed in MATLAB, specifically for this purpose. With the aiding features of the labeling tool for visualizing the sensor information, the labeling process was shortened significantly. The network was trained using the labeled data and then tested. The first designed structure, as mentioned before, had a single output with different value intervals indicating different behavior. With repeated experiments, the sensors that resulted in unnecessary noise or inconsistent measurements and that provided minimal benefit to the detection such as acceleration and step count sensors were removed from the input for better accuracy. Figure 12 shows the testing results for the trained neural network structure.
As indicated in Figure 12, different colored areas indicate different behavior. When the output is between 0.5 and 0.75, it means the pedestrian is making a right turn. If the output is between 0.5 and 0.25, the pedestrian is making a left turn. Finally, an output between 0 and 0.25 means the pedestrian is moving forward. It can be seen in the graph of Figure 12 that the prediction results (green line) are very close to ground truth (blue line) and always within the range of the correct behavior result. The second structure experimented with was created by modifying the first structure to incorporate multiple outputs to increase learning capacity, scalability, and compatibility with the literature. These outputs were then assigned as different modes of behavior, while outputs are interpreted as confidence level for this behavior. The network was trained with same data as before but labeled for multiple outputs. Test results are shown in Figure 13 that has three different outputs this time, as opposed to one output in the previous figure. Each output has confidence level between 0 and 100, where the predicted behavior should be perceived as the highest confidence level between all three of the outputs. It can be seen in the graph that the prediction (orange line) follows ground truth (blue line) closely. Although there is some noise in certain sections, prediction can successfully be observed as being correct through almost all the test data.

4.4. Pedestrian-Vehicle Interaction

The base scenario geometry for pedestrian-vehicle interaction assumes that the vehicle and the pedestrian are moving straight along paths that are perpendicular to each other. This is like one of the configurations in a recent National Highway Traffic Safety Administration (NHTSA) report on accident causes [8]. The base scenario geometry and variables are illustrated in Figure 14. A collision risk zone is defined at the crosswalk as an area of potential conflict where vehicle speed v v , pedestrian speed v p and distances to the collision zone d v and d p of the vehicle and pedestrian respectively are defined as the significant variables. This configuration was created as a base scenario to be used throughout the paper for driver warning discussions. The scenario geometry can easily be modified to include the left and right turns as well with similar variables where distance will be circular rather than linear. Using the variables mentioned above, Time to Zone (TTZ) can be calculated as
T T Z v = d v v v   T T Z p = d p v p   ,
where T T Z v and T T Z p are vehicle and pedestrian time to zones, respectively. The calculations for pedestrians can be combined with the tracking and prediction algorithm to calculate a more accurate T T Z p , especially if the pedestrian is about to turn into the crosswalk. Depending on how many seconds into the future the location is predicted, T T Z p can be completely replaced or extrapolated from prediction, using a constant velocity or similar model [31].
After the TTZ values are obtained, we can state the high collision risk condition as
T T Z p − t s < T T Z v < T T Z p + t s   ,
or
T T Z v − T T Z p < t s   ,
where t s is a safety margin time, a parameter that should be determined as a constant or dynamic value, considering the driver preferences as well as GPS accuracy. If the condition in Equations (2) or (3) is satisfied, we can say that there is a possible collision in the future. This condition is further investigated in next subsection where this correspondence is visualized, and parameters are defined and determined using real-world interaction data. As discussed above, TTZ can be calculated through defined variables as well as pedestrian path tracking and prediction. This information can then be used to determine a possible collision risk in the future, as well as to determine the severity of the condition. For each second from the start of the interaction (depicted as interaction time), there will be T T Z v and T T Z p values. An interaction diagram can be created such as in Figure 15 using the correspondence of TTZ values for the pedestrian and the vehicle.
TTZ values for the vehicle and pedestrian are shown as small colored circles as an example in Figure 15 where the color of the circle determines the interaction time of the respective TTZ values with the legend given on the right hand side vertical axis. Since both TTZ values being equal to zero, i.e. an interaction at the origin, means collision, the closer the correspondence for the ordered pair ( T T Z v , T T Z p ) gets to the origin, the more severe is the collision risk. Severity margins can be defined as circular areas around the origin for different levels of severity. Defining the severity margins is highly important for non-autonomous vehicles since the severity of the situation should be properly conveyed to the human drivers for them to react timely and correctly. Therefore, values for these margins were used as the main design parameters for the warning calculation. Another important parameter is the safety margin time, which represents the difference between TTZ values for the vehicle and the pedestrian, similar to t s that was defined in the previous subsection. If the TTZ values are reasonably different from each other, either the vehicle or the pedestrian would cross the zone first without any risk of collision. Therefore, the warning system only considers the correspondences within this safety margin time region in Figure 15.
To better visualize how the interaction graph would look with corresponding vehicle and pedestrian behavior, Figure 16 was created as an illustration. A vehicle approaching the crosswalk at the same time as a pedestrian is illustrated in Figure 16. Steps 1, 2 and 3 represent the vehicle and pedestrian locations as the vehicle is slowing down. The vehicle TTZ can be seen to be moving towards zero first but because it is slowing down, it starts to increase and goes out of the critical part of the diagram when the vehicle stops. The final image at the right side of Figure 16 shows the overall TTZ correspondence trajectory which is generally an oval arc shape.

4.5. Real World Data Processing

To gain insight from real-world interactions, data was collected and processed from a real-world intersection at Marysville, Ohio, U.S. This intersection is a smart intersection where four cameras were set up [16] to detect vehicles and pedestrians and send this information to the roadside unit (RSU) installed at the intersection. This RSU, then, broadcasted basic safety messages (BSM) for each pedestrian as well as for each vehicle with their respective location and speed information as if they are carrying a dedicated short range communication (DSRC) broadcasting device. Although this data is very large and would not be practical to physically listen to and record, the City of Marysville made it available for public use through the Smart Columbus Operating System (SCOS) database [40]. The data was downloaded and processed to extract vehicle and pedestrian interactions that fit the base scenario. This data contained eight months of BSM messages where some months had more interactions than others.
Extracted interactions were processed to calculate TTZ for interacting vehicles and pedestrians. Then, interaction graphs were created with all these interactions combined. To understand similarities within all the interaction data, a heatmap graph was also created. These graphs can be seen in Figure 17. The interaction graph on the left side of Figure 17 shows each different interaction with a different color. Since the number of interactions is very high, it is difficult to interpret this and an vehicle-pedestrian interaction density heatmap is used for better readability on the right hand side of Figure 17. The yellow area shows the densest points in the graph. When the vehicle is slowing down, the vehicle TTZ stays around similar values for a while as the pedestrian TTZ decreases steadily. The dense area in the heatmap graph corresponds to the area where vehicles are slowing down when they see a pedestrian close to the intersection. Using this information captured from real-world data allows us to design our warning system to mimic the natural behavior of drivers when they encounter nearby pedestrians.
An interesting observation looking at the behavior from real-world data comes from comparison between night and daytime interactions. Interactions were divided according to sunset time for each respective month and were plotted again Figure 18. As seen in this figure, drivers at this intersection behaved more cautiously at nighttime, resulting in an approximately 0.5 seconds shift of the dense area in the heatmap. This information can also be incorporated into safety algorithms as a preference or to provide more comfortable and cautious experience at evening and night.
Using the design parameters and real-world behavior information discussed in the previous section, driver warnings were designed for a safe stop. The warning system was designed with various severity values to convey the level of danger to the driver, hence, helping the driver’s decision on how to slow down and stop. Main criteria to determine the severity of the warning are severity margins discussed in the previous section. With the current approach that takes the TTZ as the main criteria for determining the current severity, we incorporate speed along with the remaining distance, as well as the TTZ correspondence of the pedestrian and vehicle. This prevents us from warning the driver unnecessarily in cases where either pedestrian or vehicle will pass the crosswalk without a need to stop. Three margins were defined for driver warning as Inform, Warn and Emergency with increasing degree of severity. Color coding and explanations of these margins are shown in Figure 19.
The vehicle starts with no warning even if there is a possible collision in the distant future. When the collision risk becomes closer in terms of time and T T Z v getting smaller than a certain threshold, the Inform state is issued. Inform is the lightest severity where there is a possibility for collision, but it can be avoided easily by reducing speed slowly and very comfortably. When the collision risk becomes closer and T T Z v gets smaller than a second threshold, the Warn state is issued. This means the vehicle must start slowing down as soon as possible with a relatively uncomfortable amount of deceleration to be able to stop safely. These two margins can be chosen as constant values and can afterwards be modified adaptively or by driver preference manually. To determine the value for these margins, processed real-world behavior data discussed above was used. A histogram graph of the data was used to determine a base value of 1 to 3.3 seconds for the Inform margin. After determining the base value for first margin, a Vissim simulation environment with the base scenario was constructed to test and adjust this value in simulations. Attention lag was also added to incorporate driver reaction time after noticing the warning. The test was conducted with very comfortable deceleration and stopping to obtain a value that would result in a conservative driver warning. The general flow of the simulations is illustrated in Figure 21.
As stated above, the simulations were conducted at constant start speed and time where the only changing values were random attention lag and the Inform margin value. Interaction diagrams for multiple simulation runs across different Inform margin start values are shown in Figure 21. When the diagram data was shifted by 1 sec (diagram on left), there appears to be certain cases when the vehicle cannot stop in time with comfortable braking and needs to resort to emergency braking. When it was shifted by 1.5 sec (graph in the middle of Figure 21), although there were no accidents and vehicles were able to slow down and stop comfortably, an unexpected increase in reaction time still resulted in some unwanted situations. Therefore, the value shifted by 2 sec (diagram on the right side of Figure 21) was selected as it is the most conservative one while still being close to the base value. This adjustment brought the Inform margin into the range of 3 to 5.3 seconds.
After the Inform margin is determined with both lower and upper threshold values, the Warn margin (pink area) does not need a range determination since it becomes the area between the dynamically calculated Emergency margin and the constant Inform margin. The third and final severity margin is the red Emergency margin where the vehicle has to brake immediately with harsh deceleration to be able to stop safely. Unlike threshold values defined for the previous two severities, this value is dynamic due to vehicle speed and distance being variable. Therefore, it needs to be calculated at each step to issue the driver this warning if necessary. We can calculate the minimum amount of deceleration a m i n to stop at a given distance to the conflict zone with given vehicle speed as
a min = v v 2 2 d v
If the minimum deceleration goes below the value we have for the braking deceleration limit, it means the driver must immediately press full brake to stop the vehicle. This condition will result in the red Emergency severity zone.

5. Simulation and Experimental Results and Discussion

This section discusses the evaluation and testing of the VRU safety warning system developed here. The evaluations include pedestrian path prediction evaluation, Vissim simulations, and real world testing.

5.1. Kalman Filter and LSTM Pedestrian Path Prediction Testing

As discussed previously, a system with LSTM neural networks in combination with Kalman filtering was developed for pedestrian path prediction and tracking. Test results and discussion are presented here, starting with the Kalman filter. The Kalman filter was first implemented in MATLAB and tested using real-world data recorded from an Android mobile device. Android location data from GPS and acceleration data from its accelerometer sensor were integrated together to achieve higher resolution tracking. Since the mobile device GPS updates at 1 Hz, the use of the Kalman filter is highly beneficial for the calculations as well as message broadcasting since the SAE J2735 standard messages are published at 10 Hz. Initial testing was done offline by replaying the recorded data through the algorithm. Some of the test results are plotted in Figure 22. GPS points are represented by blue circles while Kalman filter updates are represented by red cross signs. As seen in Figure 22, the filter can successfully track the pedestrian at higher frequency and smaller error. The error calculation was also done for the accuracy of the algorithm and the results were found to be satisfactory. The values are presented in Table 2 as Average Displacement Error (ADE) values in meters. After initial testing, the Kalman filtering algorithm was implemented in the Android device and was tested in a real-world setting. The results are plotted on top of a satellite image and are shown in Figure 23. As seen in the figure, the algorithm can provide higher resolution location information as compared to the Android GPS sensor while more importantly, it prevents sudden jumps in location which can be seen on the top right corner of the plots.
The LSTM neural network for pedestrian future path prediction was trained and tested offline using data collected with an Android mobile device. Pedestrian location and speed data was collected and fed to the network to test prediction for 1 s ahead first. The validation data was not included in the training dataset. Results for some of the tests are plotted in Figure 24 and Figure 25. Test data, which is the collected pedestrian path, is plotted with blue line, and the neural network predictions for each step are plotted on top as an orange line. The predictions of 1 sec into the future are very accurate for both test cases A and B. Along with the path prediction plots, calculated prediction error is presented in Table 3 as ADE. The error is smaller than that obtained with the Kalman filter.

5.2. Vissim Simulation Testing

A Vissim simulation environment was constructed with the base scenario geometry discussed above. With the vehicle speed being controllable in the simulation, several aspects of the driver warning design and slow down and stop approach were tested. First, speed information from real world data was extracted to be fed into the simulation to test the safe stop algorithm. The main idea with this simulation test was to see if the interactions obtained from the simulation are like the real world. Some differences were expected since design of the timings were made more conservatively, and deceleration was set to a very comfortable constant 0.15 g value in the simulations. At the end, the simulation structure was created with the flow illustrated in Figure 26. Vehicles will start with a random speed profile selected from a random real-world vehicle in the data, where the Inform state is issued at radial (any direction approach) 5.3 seconds. This is followed by a random attention lag, and, at the end, the vehicle will start to slow down and stop with 0.15 g deceleration.
Real world speed data along with measurement noise helped the observed behavior to be similar to the real-world data processing. Extracted interactions from simulation results are shown in Figure 27. One main feature is obvious when we look at the graph on the left compared to real world data discussed before and displayed in Figure 17. Vehicle TTZ values are not coming as close to zero as those in the graph obtained from real-world data. This behavior indicates more conservativeness. On the right side, with the heatmap graph, the area of dense points is similar to the real world counterpart, where again they are farther than the origin as compared to real data, indicating more conservative behavior for vehicles. Observing a more conservative but similar behavior in simulations shows that the fundamental approach is correct and that this method is useful for capturing the interaction and can be modified for a better fit later if desired.
All the states of Inform, Warn and Emergency were enabled in the simulation to test the design parameters with different reaction times as shown in Figure 28. Three different driver reaction times: 0.75 s, 1.25 s and 2.5 s were tested with random starting speed and random deceleration in a determined range, to see how reaction times affect the safe approach speed and what the limits are. Results are shown in Figure 29 where each graph shows multiple simulations with the vehicle starting at different speeds (y axis). The circles correspond to the severity state that caused the vehicle to stop. If the circle is gray, it means the vehicle has comfortably stopped while at the Inform state. If it is pink, stopping was a bit harder and the vehicle has stopped at the Warn state. Lastly, if there is a red circle, it means the vehicle had to perform an emergency brake and stop suddenly in order not to hit the pedestrian. With the current values for design parameters, in terms of maximum approach speed for safe stop without resorting to emergency brake, 0.75 s reaction time results in around 100 km/h, 1.25 s reaction time results in around 85 km/h and 2.5 s reaction time results in around a lower speed of 50 km/h for approaching the collision risk zone. Given that 1.25 s is a reasonable value for the average reaction time of a driver, the maximum approach speed for the current safe stop design is reasonable.

5.3. Real World Testing of Driver Warning System for Pedestrian Collision Avoidance

The mobile phone app was used to test the driver warning in the real world. The test location was selected specifically to create an NLOS scenario at an intersection with no traffic lights to demonstrate the efficacy of generating driver warnings for pedestrian collision avoidance. This scenario is illustrated on top of a satellite image of the test location in Figure 30. The vehicle starts around the left edge of the garage building and moves towards the intersection following the blue arrow while the pedestrian starts behind the building and away from the intersection following the green arrow. The collision risk zone is also illustrated and is used to calculate TTZ values. Two phones running the application were used in the test, one in the vehicle and the other one carried by the pedestrian. Along with the data recording inside the phone for communication and warning calculation data, a camera was fixed inside the vehicle to record the part of the driver view and the head-up display (HUD) where the warning was displayed. Snapshots from the recorded video can be seen in Figure 31.
As seen in Figure 31, a HUD was fixed in front of the driver where the phone is placed on with the phone screen facing upward. Reflection of the screen on the HUD glass can be observed by the driver clearly although it is a bit faded in the figures. The numbering of 1 to 4 is according to chronological order of the events, corresponding to the vehicle getting closer to the intersection. The vehicle starts to drive with no warning in picture 1. Since the phone is receiving the V2P transmission even with no line of sight, as it can be seen in picture 2, the driver receives a yellow-colored warning, indicating the Inform state and that there is a pedestrian approaching towards the intersection. In picture 3, the vehicle comes closer to the intersection without slowing down and the warning turns to orange to indicate increased severity, representing the Warn state. Finally, in picture 4, the vehicle stops safely, and the pedestrian shows up on the front window view (right hand side) while starting to cross in front of the vehicle. The HUD window is clear with no warnings since the vehicle has already stopped. Hence, the real-world testing was successfully completed for the application. The data recorded from the phone was also visualized in the summary view of Figure 32.
The garage building can be seen as the gray rectangle in the middle of Figure 32 where vehicle and pedestrian positions are plotted using color coded warnings corresponding to those positions. The light blue color represents the pedestrian, and the vehicle is represented by the dark blue color when there is no warning. When there is a warning, the vehicle location is indicated by the color of the warning as yellow (Inform) and orange (Warn) circles on the graph. The sequence of events can be observed in detail in the figure. The vehicle starts at the top left with no warning while the pedestrian starts at bottom right. The location of points represents the GPS measurements as the vehicle was receiving the V2P messages. A while after the vehicle starts moving, the driver receives a yellow warning while the pedestrian is still behind the garage building, resulting in NLOS. Following the yellow warning, an orange warning is issued while the vehicle gets closer to the collision zone without slowing down. At the end, the vehicle stops safely, and the warning is disabled, and this location can be seen as a dark blue circle. The pedestrian continues to move and crosses in front of the stopped vehicle.

6. Conclusions and Recommendations

A V2P communication-based driver warning system for pedestrian safety was developed and evaluated in this paper. The paper also considered possible problems and derived methods to remedy these problems. These methods were evaluated in several ways including realistic simulation environments and real-world testing. By utilizing mobile phone sensors and Bluetooth communication on the Android smartphones, a V2P safety application was developed to broadcast information for pedestrian motion while providing warnings to the human driver, when necessary, with the developed driver warning system. This application allows easy access to a pedestrian safety system for public use. The developed application was also used for data collection from the sensors on the Android phone. With these sensors, capabilities of the data collection application, as well as virtual environments created in Vissim, neural network structures were trained and experimented with for pedestrian path and behavior predictions. Kalman Filtering was also utilized for higher frequency pedestrian motion tracking.
One of the key contributions of the paper was utilization of real-world data to create a baseline for driver safety warning system design parameters, with a goal of leading to a more natural behavior for the safety system user. Besides this, some interesting observations were made while comparing the driver behavior in terms of several aspects such as approach direction or time of day. It was observed that drivers were more cautious while approaching the intersection with the pedestrian at times of the day where sunlight is limited or not available. The data analyzed and used had a span of several months and included a large amount of BSM messages that were published for all vehicles and pedestrians approaching towards or crossing the street at the chosen smart intersection as well as signal phase and timing (SPaT) messages.
Among current limitations that need to be explored further in detail is how to treat crowds of pedestrians (VRUs). Although the way the application is currently designed to work will not create problems in terms of providing a warning when there are multiple pedestrians, problems may arise due to highly increased processing load and congestion in Bluetooth communication band. Another limitation of the system that should be mentioned is the GPS accuracy of the Android phones. Although the accuracy was satisfactory enough to provide reliable data and warnings in many data collection and warning system experiments, this might not always be the case. The dynamics of the vehicle [41] on a road may also affect GPS accuracy. However, the Kalman Filter in the current implementation helps with this issue, especially for irregular jumps in the GPS location. One of the expected improvements in the near future is dual-frequency GPS. As the dual-frequency GPS smartphones get more common, the GPS accuracy issue will be reduced over time. On top of this, RTK GPS chips are getting smaller and cheaper over time while providers are building stations to publish RTK correction messages over the internet for better accessibility. With the RTK GPS technology getting wider availability, GPS accuracy for small devices like smartphones will be significantly improved. This will help in the implementation success of the V2P communication based pedestrian safety system presented here.
While this paper focused on a driver warning system, a collision avoidance system with collision free trajectory computations and trajectory tracking control [3] are also needed and will be the focus of future work. Parameter space based robust control methods [42], [43] can be used for robust trajectory tracking control for a collision mitigation system for rejection of disturbances in the presence of model uncertainty. Future work can also focus on the fused use of V2P communication and AV perception sensors [44], and the use of socially acceptable VRU collision avoidance in low-speed campus like deployments [45].

Author Contributions

Conceptualization, S.Y.G., B.A.-G. and L.G.; methodology, S.Y.G., B.A.-G. and L.G.; software, S.Y.G.; validation, S.Y.G.; formal analysis, S.Y.G.; investigation, S.Y.G., B.A.-G. and L.G.; resources, B.A.-G. and L.G.; data curation, S.Y.G.; writing—original draft preparation, S.Y.G.; writing—review and editing, B.A.-G. and L.G.; visualization, S.Y.G.; supervision, B.A.-G. and L.G.; project administration, B.A.-G. and L.G.; funding acquisition, B.A.-G. and L.G. All authors have read and agreed to the published version of the manuscript.

Funding

This research was partially funded by the Ford Alliance program.

Data Availability Statement

Data are contained within the article.

Acknowledgments

The authors thank the Automated Driving Lab at the Ohio State University. The authors thank Gopichandra Surnilla and Jay Parikh for insightful discussions on V2P communication and VRU safety.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. "Early Estimate of Motor Vehicle Traffic Fatalities in 2018," NHTSA, 2019.
  2. Guvenc, L.; Guvenc, B.A.; Emirler, M.T. Connected and Autonomous Vehicles. In Internet of Things and Data Analytics Handbook; John Wiley & Sons, Ltd.: Hoboken, NJ, USA, 2017; pp. 581–595, ISBN: 978-1-119-17360-1.
  3. Guvenc, L.; Aksun-Guvenc, B.; Zhu, S.; Gelbal, S.Y. Autonomous Road Vehicle Path Planning and Tracking Control; Book Series on Control Systems Theory and Application; Wiley/IEEE Press: New York, NY, USA, 2021; ISBN 978-1-119-74794-9. [Google Scholar]
  4. Gelbal, S.Y.; Aksun-Guvenc, B.; Guvenc, L. SmartShuttle: A Unified, Scalable and Replicable Approach to Connected and Automated Driving in a Smart City. In Proceedings of the Science of Smart City Operations and Platforms Engineering in partnership with Global City Teams Challenge (SCOPE-GCTC) Workshop, Pittsburgh, PA, USA, 18–21 April 2017. [Google Scholar]
  5. M. Bagheri, M. Siekkinen and J. K. Nurminen, "Cloud-Based Pedestrian Road-Safety with Situation-Adaptive Energy-Efficient Communication," IEEE Intelligent Transportation Systems Magazine, vol. 8, no. 3, pp. 45-62, 2016. [CrossRef]
  6. "WHO - Advice for the public," [Online]. Available: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public#:~:text=Maintain%20at%20least%20a%201,The%20further%20away%2C%20the%20better.. [Accessed November 2020].
  7. T. Gandhi and M. M. Trivedi, "Pedestrian Protection Systems: Issues, Survey, and Challenges," IEEE Transactions on Intelligent Transportation Systems, vol. 8, no. 3, pp. 413-430, 2007.
  8. E. Swanson, M. Yanagisawa, W. G. Najm, F. Foderaro and P. Azeredo, "Crash Avoidance Needs and Countermeasure Profiles for Safety Applications Based on Light-Vehicle-to-Pedestrian Communications," National Highway Traffic Safety Administration (NTHSA), Washington, DC, 2016.
  9. K. Hickey, "NYC looks to tech to improve pedestrian safety," 2016. [Online]. Available: https://gcn.com/articles/2016/10/19/nyc-ped-sig.aspx. [Accessed 10 12 2021].
  10. Marco Dozza, Julia Werneke, “Introducing naturalistic cycling data: What factors influence bicyclists’ safety in the real world?,” Transportation Research Part F: Traffic Psychology and Behaviour, Volume 24, 2014, Pages 83-91, ISSN 1369-8478. [CrossRef]
  11. Qingyu Ma, Hong Yang, Alan Mayhue, Yunlong Sun, Zhitong Huang, Yifang Ma, “E-Scooter safety: The riding risk analysis based on mobile sensing data,” Accident Analysis & Prevention, Volume 151, 2021, 105954, ISSN 0001-4575. [CrossRef]
  12. NTHSA, "Vehicle-To-Vehicle Communication Technology for Light Vehicles," 2019.
  13. J.-K. Bae, M. -C. Park, E. -J. Yang and D. -W. Seo, "Implementation and Performance Evaluation for DSRC-Based Vehicular Communication System," in IEEE Access, vol. 9, pp. 6878-6887, 2021. [CrossRef]
  14. S. Gyawali, S. Xu, Y. Qian and R. Q. Hu, "Challenges and Solutions for Cellular Based V2X Communications," in IEEE Communications Surveys & Tutorials, vol. 23, no. 1, pp. 222-255, Firstquarter 2021. [CrossRef]
  15. E. Moradi-Pari, D. Tian, M. Bahramgiri, S. Rajab and S. Bai, "DSRC Versus LTE-V2X: Empirical Performance Analysis of Direct Vehicular Communication Technologies," in IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 5, pp. 4889-4903, May 2023. [CrossRef]
  16. Honda, "Honda Demonstrates New "Smart Intersection" Technology that Enables Vehicles to Virtually See Through and Around Buildings," 2018. [Online]. Available: https://hondanews.com/en-US/releases/honda-demonstrates-new-smart-intersection-technology-that-enables-vehicles-to-virtually-see-through-and-around-buildings. [Accessed 10 12 2021].
  17. E. Moradi-Pari, D. Tian, H. N. Mahjoub and S. Bai, "The Smart Intersection: A Solution to Early-Stage Vehicle-to-Everything Deployment," in IEEE Intelligent Transportation Systems Magazine, vol. 14, no. 5, pp. 88-102, Sept.-Oct. 2022. [CrossRef]
  18. P. Merdrignac, O. Shagdar and F. Nashashibi, "Fusion of Perception and V2P Communication Systems for the Safety of Vulnerable Road Users," in IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 7, pp. 1740-1751, July 2017. [CrossRef]
  19. C. Gentner, D. Günther and P. H. Kindt, "Identifying the BLE Advertising Channel for Reliable Distance Estimation on Smartphones," in IEEE Access, vol. 10, pp. 9563-9575, 2022. [CrossRef]
  20. Y. -R. Tsai and Y. -C. Chen, "Opportunistic Connectionless Undirected Information Dissemination Based on Bluetooth Low Energy Advertising Technology on Smartphones," in IEEE Access, vol. 9, pp. 155851-155860, 2021. [CrossRef]
  21. M. Wu, B. Ma, Z. Liu, L. Xiu and L. Zhang, "BLE-horn: A smartphone-based bluetooth low energy vehicle-to-pedestrian safety system," 2017 9th International Conference on Wireless Communications and Signal Processing (WCSP), Nanjing, China, 2017, pp. 1-6. [CrossRef]
  22. Y. Wang et al., "Determining Driver Phone Use by Exploiting Smartphone Integrated Sensors," in IEEE Transactions on Mobile Computing, vol. 15, no. 8, pp. 1965-1981, 1 Aug. 2016. [CrossRef]
  23. S.-E. Ramah, A. Bouhoute, K. Boubouh and I. Berrada, "One Step Further Towards Real-Time Driving Maneuver Recognition Using Phone Sensors," in IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 10, pp. 6599-6611, Oct. 2021. [CrossRef]
  24. Google, "Android Bluetooth Low Energy Documentation," [Online]. Available: https://developer.android.com/guide/topics/connectivity/bluetooth/ble-overview. [Accessed 10 12 2021].
  25. E. U. Warriach and S. Witte, "Approach for performance investigation of different Bluetooth modules and communication modes," 2008 4th International Conference on Emerging Technologies, Rawalpindi, Pakistan, 2008, pp. 167-171. [CrossRef]
  26. Y. Zhang, X. Wang, K. Zhuo, W. Jiao and W. Yang, "Research on pedestrian vehicle collision warning based on path prediction," 2023 7th International Conference on Transportation Information and Safety (ICTIS), Xi'an, China, 2023, pp. 1267-1272. [CrossRef]
  27. G. Xiog, T. Yang, M. Li, Y. Zhang, W. Song and J. Gong, "A Novel V2X-based Pedestrian Collision Avoidance System and the Effects Analysis of Communication Delay and Packet Loss on Its application," 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES), Madrid, Spain, 2018, pp. 1-6. [CrossRef]
  28. Oncu Ararat, Bilin Aksun Guvenc, “Development of a Collision Avoidance Algorithm Using Elastic Band Theory,” IFAC Proceedings Volumes, Volume 41, Issue 2, 2008, Pages 8520-8525, ISSN 1474-6670, ISBN 9783902661005. [CrossRef]
  29. R. Quintero Mínguez, I. Parra Alonso, D. Fernández-Llorca and M. Á. Sotelo, "Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition," in IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 5, pp. 1803-1814, May 2019. [CrossRef]
  30. S. Min, J.-Y. Lee and K.-D. Jung, "Real-time Path Prediction and Grid-based Path Modeling Method," International Journal of Applied Engineering Research, vol. 12, no. 20, pp. 9997-10001, 2017.
  31. C. Scholler, V. Aravantinos, F. Lay and A. Knoll, "What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 1696-1703, 2020. [CrossRef]
  32. A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei and S. Savarese, "Social LSTM: Human Trajectory Prediction in Crowded Spaces," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  33. Gupta, J. Johnson, L. Fei-Fei, S. Savarese and A. Alahi, "Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  34. H. Xue, D. Huynh and M. Reynolds, "Location-Velocity Attention for Pedestrian Trajectory Prediction," in IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
  35. H. W. Sorenson, Kalman Filtering: Theory and Application, IEEE Press, 1985.
  36. "nrf52840-DK Bluetooth Development Board," Nordic Semiconductor, [Online]. Available: https://www.nordicsemi.com/Products/Development-hardware/nrf52840-dk. [Accessed 10 12 2021].
  37. SAE, "J2735 Dedicated Short Range Communications (DSRC) Message Set Dictionary," 2016.
  38. "WIZnet W5500 Ethernet Shield," [Online]. Available: https://www.wiznet.io/product-item/w5500-ethernet-shield/. [Accessed 10 12 2021].
  39. T. Kulshrestha, R. Niyogi and D. Patel, "A fast and scalable crowd sensing based trajectory tracking system," 2017 Tenth International Conference on Contemporary Computing (IC3), Noida, India, 2017, pp. 1-6. [CrossRef]
  40. "SCOS Database," [Online]. Available: https://www.smartcolumbusos.com/. [Accessed 10 12 2021].
  41. Ozcan, D.; Sonmez, U.; Guvenc, L. Optimisation of the Nonlinear Suspension Characteristics of a Light Commercial Vehicle. Int. J. Veh. Technol. 2013, 2013, 562424. [CrossRef]
  42. Guvenc, L.; Aksun-Guvenc, B.; Demirel, B.; Emirler, M. Control of Mechatronic Systems; IET: London, UK, 2017. [Google Scholar]
  43. Necipoglu, S., Cebeci, S.A., Has, Y.E., Guvenc, L., Basdogan, Ç., 2011, “A Robust Repetitive Controller for Fast AFM Imaging,” IEEE Transactions on Nanotechnology, Vol. 10, No. 5, pp. 1074-1082. [CrossRef]
  44. Bowen, W., Gelbal, S.Y., Aksun-Güvenç, B., Güvenç, L., 2018, “Localization and Perception for Control and Decision Making of a Low Speed Autonomous Shuttle in a Campus Pilot Deployment,” SAE International Journal of Connected and Automated Vehicles. https://doi.org/10.4271/12-01-02-0003, Vol. 1, Issue 2, pp. 53-66. [CrossRef]
  45. M.T. Emirler; H. Wang; Bilin Aksun-Guvenc, “Socially Acceptable Collision Avoidance System for Vulnerable Road Users,” IFAC-PapersOnLine, Volume 49, Issue 3, 2016, Pages 436-441, ISSN 2405-8963. [CrossRef]
Preprints 92296 g001
Figure 1. Illustration of V2P communication over Bluetooth.
Figure 1. Illustration of V2P communication over Bluetooth.
Preprints 92296 g001
Preprints 92296 g002
Figure 2. PSM over Bluetooth V2P app running on a smartphone.
Figure 2. PSM over Bluetooth V2P app running on a smartphone.
Preprints 92296 g002
Preprints 92296 g003
Figure 3. The nRF52840-DK board with W5500 Ethernet shield mounted.
Figure 3. The nRF52840-DK board with W5500 Ethernet shield mounted.
Preprints 92296 g003
Preprints 92296 g004
Figure 4. PSM over Bluetooth V2P app process flow diagram.
Figure 4. PSM over Bluetooth V2P app process flow diagram.
Preprints 92296 g004
Preprints 92296 g005
Figure 5. Data collection app running in the smartphone.
Figure 5. Data collection app running in the smartphone.
Preprints 92296 g005
Preprints 92296 g006
Figure 6. Collected data plotted in MATLAB.
Figure 6. Collected data plotted in MATLAB.
Preprints 92296 g006
Preprints 92296 g007
Figure 7. Pedestrian safety app running on a smartphone.
Figure 7. Pedestrian safety app running on a smartphone.
Preprints 92296 g007
Preprints 92296 g008
Figure 8. Architecture of the overall safety approach.
Figure 8. Architecture of the overall safety approach.
Preprints 92296 g008
Preprints 92296 g009
Figure 9. Pedestrian tracking and path prediction using LSTM neural networks and a Kalman filter.
Figure 9. Pedestrian tracking and path prediction using LSTM neural networks and a Kalman filter.
Preprints 92296 g009
Preprints 92296 g010
Figure 10. Some of the samples used for training the LSTM neural networks.
Figure 10. Some of the samples used for training the LSTM neural networks.
Preprints 92296 g010
Preprints 92296 g011
Figure 11. Pedestrian behavior prediction initial structure.
Figure 11. Pedestrian behavior prediction initial structure.
Preprints 92296 g011
Preprints 92296 g012
Figure 12. Single output prediction results.
Figure 12. Single output prediction results.
Preprints 92296 g012
Preprints 92296 g013
Figure 13. Multi-output prediction results.
Figure 13. Multi-output prediction results.
Preprints 92296 g013
Preprints 92296 g014
Figure 14. Geometrical configuration and variables for base scenario.
Figure 14. Geometrical configuration and variables for base scenario.
Preprints 92296 g014
Preprints 92296 g015
Figure 15. Pedestrian (VRU) and vehicle interaction diagram.
Figure 15. Pedestrian (VRU) and vehicle interaction diagram.
Preprints 92296 g015
Preprints 92296 g016
Figure 16. Illustration of vehicle and pedestrian behavior with successive interaction diagrams.
Figure 16. Illustration of vehicle and pedestrian behavior with successive interaction diagrams.
Preprints 92296 g016
Preprints 92296 g017
Figure 17. Nearby vehicle and pedestrian interaction graph (left) and heatmap (right) based on real-world data.
Figure 17. Nearby vehicle and pedestrian interaction graph (left) and heatmap (right) based on real-world data.
Preprints 92296 g017
Preprints 92296 g018
Figure 18. Comparison of daylight and no daylight interactions.
Figure 18. Comparison of daylight and no daylight interactions.
Preprints 92296 g018
Preprints 92296 g019
Figure 19. Warning severities and meanings.
Figure 19. Warning severities and meanings.
Preprints 92296 g019
Preprints 92296 g020
Figure 20. Flow of Vissim simulation to adjust the Inform margin.
Figure 20. Flow of Vissim simulation to adjust the Inform margin.
Preprints 92296 g020
Preprints 92296 g021
Figure 21. Simulation results for adjusted Inform margin (gray area) values. The color coded severity areas are for default values assigned before the Vissim simulation study. The gray area for the inform margin is afterwards adjusted for the most conservative case on the right subplot to be between 3 sec and 5.3 sec.
Figure 21. Simulation results for adjusted Inform margin (gray area) values. The color coded severity areas are for default values assigned before the Vissim simulation study. The gray area for the inform margin is afterwards adjusted for the most conservative case on the right subplot to be between 3 sec and 5.3 sec.
Preprints 92296 g021
Preprints 92296 g022
Figure 22. Kalman Filter testing with Android mobile phone data.
Figure 22. Kalman Filter testing with Android mobile phone data.
Preprints 92296 g022
Preprints 92296 g023
Figure 23. Kalman filter real-world testing with Android mobile device.
Figure 23. Kalman filter real-world testing with Android mobile device.
Preprints 92296 g023
Preprints 92296 g024
Figure 24. LSTM pedestrian path prediction test A.
Figure 24. LSTM pedestrian path prediction test A.
Preprints 92296 g024
Preprints 92296 g025
Figure 25. LSTM pedestrian path prediction test B.
Figure 25. LSTM pedestrian path prediction test B.
Preprints 92296 g025
Preprints 92296 g026
Figure 26. Simulation flow for Vissim interaction testing.
Figure 26. Simulation flow for Vissim interaction testing.
Preprints 92296 g026
Preprints 92296 g027
Figure 27. Vissim pedestrian-vehicle interaction testing.
Figure 27. Vissim pedestrian-vehicle interaction testing.
Preprints 92296 g027
Preprints 92296 g028
Figure 28. Simulation flow for Vissim driver reaction time testing.
Figure 28. Simulation flow for Vissim driver reaction time testing.
Preprints 92296 g028
Preprints 92296 g029
Figure 29. Simulation results for different driver reaction times.
Figure 29. Simulation results for different driver reaction times.
Preprints 92296 g029
Preprints 92296 g030
Figure 30. Real world test scenario illustration.
Figure 30. Real world test scenario illustration.
Preprints 92296 g030
Preprints 92296 g031
Figure 31. In vehicle snapshots recorded during testing.
Figure 31. In vehicle snapshots recorded during testing.
Preprints 92296 g031
Preprints 92296 g032
Figure 32. Visualization of communication and warning data during the testing.
Figure 32. Visualization of communication and warning data during the testing.
Preprints 92296 g032
Table 1. Bluetooth communication range experiments.
Table 1. Bluetooth communication range experiments.
Device Condition Distance in Normal Advertisement (m) Distance in Extended Advertisement (m)
nRF52840-DK No Obstructions 191 253
Android No Obstructions 78 114
Android Intentionally Obstructed by Pedestrian Body 27 55
Table 2. Tracking error for Kalman filter testing.
Table 2. Tracking error for Kalman filter testing.
Test ADE (m)
A 0.2866
B 0.2112
C 0.3109
Table 3. Prediction error for LSTM.
Table 3. Prediction error for LSTM.
Test ADE (m)
A 0.1132
B 0.1556
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated