Blog

09/08/2023 by ardi.tampuu

ADL Minicar Challenge 2023: the organizer's perspective

In this blog post, we try to illustrate what was going on behind the scenes of the ADL MInicar Challenge 2023.

ADL Minicar challenge is a self-driving software competition. This means that while the competing solutions will be deployed and must perform well on hardware, modifications to the hardware are forbidden. The competition was held in January 2021 and January 2022 as part of the DeltaX student competition series but was hosted independently in 2023 under the flag of the Autonomous Driving Lab. Video from the first competition in the year 2021 is added here, videos from 2022 can be seen below.

If you want to see the final results of the 2023 challenge, achieved by the participants, you must scroll all the way to the bottom of the page. The tasks and challenges the organizers and participants had to face when organizing the event are reported in chronological order, to the best recollection of the author.

Designing the task and rules for 2023
First of all, it is important to note that the competition is not a one-day event, but preparations for it last for an entire semester.

To illustrate:

• In August-September, the competition rules were developed. The track was designed.
• In September, the competition was advertised by a promo video running on a big screen 24/7 for two weeks. The competition was advertised to students in multiple courses.
• In September, the vehicles were all tested and put in working order
• In October-December, there were regular meetings with the teams participating in the competition as part of the Autonomous Vehicles Project course. Four out of five other teams also had regular contact and meetings.
• In December, organizers produced videos exemplifying the scoring logic on trial runs
• In early January, the organizers held a mock competition to estimate how long everything takes in practice, and how much time must be reserved for the competition.
•  Competition day 25th of January 2023
•  Verifying validity of winning solutions, and paying out rewards.

In a similar competition back in January 2022, the students had competed in and solved the task of obstacle avoidance and route following in a town with an unknown road layout and object locations. Essentially, the teams knew the streets would be made of the wooden walls we have at the track, streets would be 60+ cm wide, and Lego cars, people, and animals may be placed in random locations on this track. The models were expected to generalize to a wide variety of situations. Beyond that, students had to struggle with different light conditions and effective speed (speed decreases as the battery empties and engines heat up) messing up the performance. 

And the students prevailed.

This winning performance was in fact achieved with a solution that was computationally unnecessarily costly, with every frame taking about 120-130 milliseconds to deal with. This results in 7-8 frames being processed per second. During the competition, it was visible that this solution was slow to react to obstacles compared to the eventual second-place solution. The big difference in reactiveness, decisions-per-second, came from using a slow .h5 model format instead of .tflite. The faster model did not win, simply as it was trained on considerably less data (10K frames instead of 150K).

Relevant for designing the 2023 event:

•  A slow, not optimal solution solved the obstacle avoidance problem and a faster implementation would likely do even better in obstacle avoidance.

As an additional piece of information:

• A student project in spring 2022 (link) demonstrated that the right-hand rule can be taught to DonkeyCars

We concluded that in the 2023 competition, more complete urban driving should be attempted - avoiding walls, and Lego characters, as well as following some traffic rules. For this, we needed a toy town where all this could take place. A 1:10 scale town, as the donkey cars are 1:10 scale. A few blocks of houses. Do you see the problem? Where can we find the space in the University of Tartu Delta building to build this town? Every street is about 8 meters wide in the real world. Every building is measured in tens of meters.

A large town would allow many different trajectories and require more generalization from the participants. But being restricted by space, we ended up with the following town.

From START, the vehicle must either go right or left at the T-junction according to the traffic sign. It must then complete a lap safely, without collisions, by following rules on the road crossing and the other T-junction where there might be another vehicle. When a  stop sign is placed on the road the vehicle must stop.

Rules development
In the 2021 competition (rules link) we judged solutions by driving clean + faster speed, and in 2022 by driving without collisions in increasingly complicated situations (speed didn't matter). Here, we could have also just counted infractions, but running over a pedestrian on a road crossing feels like a more severe violation than touching the wall with a wheel. We wanted a scoring scheme that would suit the use case of urban driving.

Luckily benchmarks of urban driving exist in simulation. We took inspiration from the CARLA benchmark and decided to penalize different infractions differently, as well as count the proportion of the route the model managed to complete autonomously.Many questions arose.

1. If the model gets stuck, is the attempt over, or will the "safety driver" intervene and lift the vehicle back on the track? Where on the track? Before the failed turn or after? Lifting the vehicle forward would help the solution clear the challenge immediately and would not allow separating teams that never succeed in some aspect from teams that fail only from time to time. But if the model fails the same turn repeatedly, how many times will it be lifted back and can try again?  
2. If a car can reverse itself out of a failed situation and solve it without safety-driver intervention, should it be penalized less than a crash+ getting stuck?
3. If in the same accident, a car, pedestrian, and wall are hit, will the penalty be the product of these different infractions?
4. If we say speed doesn't matter, only safe driving, can a vehicle spend half an hour on one lap? Is stopping to analyze the situation allowed at all?

We found reasonable compromises, wrote the rules (rules link), and even made a set of example runs with scoring illustrated.


The students were advised to (but were free to solve it otherwise):

• Start with general object avoidance ability, so they would get something running
• Consider driving at a constant throttle and only controlling the steering
• Make solutions for "stopping conditions" - stop sign, human on the road crossing, right-hand rule. So these simply set the otherwise constant throttle to 0

For example:

The final event 25th of January 2023
In the wake of the final competition day, out of eight teams five dropped out and decided not to participate:

• A very strong team, as per the videos shown to the organizers, had been down with Corona for a month and didn't have enough time to prepare.
• One team had tried to take a Reinforcement Learning approach in simulation and then transfer the model to the track. This didn't work out (but they learned a lot).
• One team of exchange students left Estonia before the final event
• One team was burned out with fall semester work and decided not to take extra obligations in January
• One team did not reply about their reasons

As an organizer, having solved technical issues and guided the students for months, it's slightly disappointing. At the same time, it is easier to host an event for 3 teams, instead of 8. Simply the time to run repeated trials for each solution takes a long time. We either have all teams run the same scenario and only then switch the object locations, or we have one team run all their runs in a row. The first option means a lot of time is lost between runs, as switching teams takes time. The latter means organizers must change the layout of signs and obstacles, attentively, 24 times for 3 teams. With only 3 teams we chose the former.

For everything to run smoothly, there were many judges and assistants:

• Ardi and Kertu acted as main judges, calling out all infractions, changing the layout of obstacles, making decisions 
• Aral noted down all infractions the judges called in a table that automatically calculated scores (see below the marked down infractions for the best scoring run with a score of 0.206)
• Tambet noted down the exact versions of code the participants used, by verifying the Git remote address and version number. This allowed a later check to make sure there was nothing illegal in the code
• Edgar and Mahir were filming each attempt with their phones from two different angles. We verified the scores, recounting infractions, from those videos during and after the event.
• Laima kept track of the attempts and team numbers and showed this to cameras before every attempt 
• Britta from ATI comp support team to set up static cameras at opposite angles of the track
• Karl from the industry collaboration team to conduct interviews with participants

The order of teams was randomly drawn and the competition consisting of 8 scenarios, designed beforehand by Ardi and Kertu, proceeded in order Scenario1-Team1, Scenario1-Team2, Scenario1-Team3, (reconfigure track), Scenario2-Team1,...

It was quickly clear that the performance the teams expected and had experienced in their prior trials alone at the track, did not realize. In fact:

• The teams reacted to the direction of the driving sign with less than 50% accuracy. Whenever the car turned even slightly in the right direction, even if it ended up in the wall, this penalty was not applied. Furthermore, one team had implemented stopping 1 second into the drive to see the sign more clearly, in higher resolution to detect the arrowhead direction. Still, the detection or the mechanism to react to the detection failed.
• All teams had decided to use Ni-H2 batteries, instead of the larger capacity Lithium Ion batteries. Maybe due to this, they struggled to reliably set the vehicle speed in a range suitable for their solutions. In multiple cases, the vehicle did not start moving or stopped mid-route as the voltage output was no longer sufficient for driving (due to higher friction in turns, battery depletion, or engine heating). Alternatively, in a few cases, the vehicle drove too fast for its own good. While maintaining good speed was an issue in the 2022 competition, it was not the decisive factor. Somehow in 2023, the problem was more severe. The choice of batteries might have been the cause.
• Solutions tended to detect STOP signs where there were none. A spectator with a red shirt was likely the cause. The stop sign was placed so high by the organizers that the solutions couldn't cut away the top half of the image to get rid of the noisy background in their models. Probably this noise hadn't contained red stuff in the training data.

• Vehicles were at times a bit lazy in the turns, not turning steep. This caused trouble in the following straight or next turn. E.g. hitting obstacles placed on the outside curve; ending up in the park with animals after the zig-zag turn; ending the U-turn unusually late and not reacting well to the pedestrian on the crossing.
• One team had struggled with their car losing its turning radius and had to swap cars. Different cars are somewhat different in their turning properties. Also, the Raspberry Pi camera may be set at a different focal length or may look slightly higher or lower. Neural networks are very sensitive to such slight changes.

Due to these issues, and possibly just the general ability of the models to generalize to different backgrounds and light conditions, the scores of the participating teams remained low. The highest score for an individual route was 0.206, and it contained a variety of smaller infractions. The authors of this solution were Mihkel Lepson and Artur Tuttar.

However, the winning criterion was the best average of 8 runs. The above team scored an average 0.054 driving score, whereas Yevhen Pankevych and Volodymyr Savchuk reached an average score of 0.068. Notice that with 8 laps, one perfectly clean attempt would add 0.125 points to the average and would have won the competition by a margin.

The winners scored more consistently, their best run is given below.

Results of the 2023 competition:

1st place (1000 euros prize): Yevhen Pankevych and Volodymyr Savchuk, score 0.068, GitHub
2nd place (500 euros prize): Mihkel Lepson and Artur Tuttar, score 0.054, GitHub
3rd place (250 euros prize): Pavlo Pyvovar, score 0.020, GitHub

Closing remarks
The competition proved useful in teaching students that deployment is the longest and most complicated phase of any data science solution. They got first-hand experience in curating a dataset to improve results at deployment time, deploying and improving iteratively. These are a few of the lessons students mentioned in their interviews, which can be watched here: (folder).

As for the organizers, we learned our scoring was suited for the strong performance we expected, but struggled to be useful in the presence of 6+ infractions per lap. The event took longer than we expected, even though we considered 5 minutes per run, with driving taking about 1 minute maximally. In the year 2024 competition, we will improve these aspects, because otherwise hosting an event with more than 3 teams will be challenging.
Also, the students will be advised to solve the overall task, starting from the challenges the vehicle encounters first on the track so that the driving direction sign will be solved better.