Writeup Template: You can use this file as a template for your writeup if you want to submit it as a markdown file, but feel free to use some other method and submit a pdf if you prefer.
The goals / steps of this project are the following:
Training / Calibration
- Download the simulator and take data in "Training Mode"
- Test out the functions in the Jupyter Notebook provided
- Add functions to detect obstacles and samples of interest (golden rocks)
- Fill in the
process_image()function with the appropriate image processing steps (perspective transform, color threshold etc.) to get from raw images to a map. Theoutput_imageyou create in this step should demonstrate that your mapping pipeline works. - Use
moviepyto process the images in your saved dataset with theprocess_image()function. Include the video you produce as part of your submission.
Autonomous Navigation / Mapping
- Fill in the
perception_step()function within theperception.pyscript with the appropriate image processing functions to create a map and updateRover()data (similar to what you did withprocess_image()in the notebook). - Fill in the
decision_step()function within thedecision.pyscript with conditional statements that take into consideration the outputs of theperception_step()in deciding how to issue throttle, brake and steering commands. - Iterate on your perception and decision function until your rover does a reasonable (need to define metric) job of navigating and mapping.
Rubric Points
Here I will consider the rubric points individually and describe how I addressed each point in my implementation.
1. Provide a Writeup / README that includes all the rubric points and how you addressed each one. You can submit your writeup as markdown or pdf.
You're reading it!
1. Run the functions provided in the notebook on test images (first with the test data provided, next on data you have recorded). Add/modify functions to allow for color selection of obstacles and rock samples.
After running the code on test data, I was able to obtain results as in the following screenshots :
For detecting rock samples a seperate function is written as follows.
def find_rocks(img, levels=[110,110,50]):
rockpix = ((img[:,:,0] > levels[0]) \
& (img[:,:,1] > levels[1]) \
& ( img[:,:,2] < levels[2]))
color_select = np.zeros_like(img[:,:,0])
color_select[rockpix] = 1
return color_select`The rock color mainly stays in the red, green levels and stays less in blue. So modified the color selection function accordingly. The result is as follows :
1. Populate the process_image() function with the appropriate analysis steps to map pixels identifying navigable terrain, obstacles and rock samples into a worldmap. Run process_image() on your test data using the moviepy functions provided to create video output of your result.
Modified Jupyter Notebook to accomodate the required changes. All details are shown in the jupyter notebook 'Rover_Project_Test_Notebook.ipynb'. Test Video Included .
1. Fill in the perception_step() (at the bottom of the perception.py script) and decision_step() (in decision.py) functions in the autonomous mapping scripts and an explanation is provided in the writeup of how and why these functions were modified as they were.
def perspect_transform(img, src, dst):
M = cv2.getPerspectiveTransform(src, dst)
warped = cv2.warpPerspective(img, M, (img.shape[1], img.shape[0]))# keep same size as input image
mask = cv2.warpPerspective(np.ones_like(img[:,:,0]),M,(img.shape[1],img.shape[0]))
return warped,maskUsed Perspective Transform along with a mask to detect part of the image which is in the visible region of the camera and added the function to find the rocks. Since obstacle map is used along with the mask being applied, I was able to improve the mapping. Each of the steps are carried out as in the commented in the code. If any rock was found it is marked in the map.
def perception_step(Rover):
# Perform perception steps to update Rover()
# TODO:
# NOTE: camera image is coming to you in Rover.img
# 1) Define source and destination points for perspective transform
dst_size = 5
bottom_offset = 6
image = Rover.img
source = np.float32([[14, 140], [301 ,140],[200, 96], [118, 96]])
destination = np.float32([[image.shape[1]/2 - dst_size, image.shape[0] - bottom_offset],
[image.shape[1]/2 + dst_size, image.shape[0] - bottom_offset],
[image.shape[1]/2 + dst_size, image.shape[0] - 2*dst_size - bottom_offset],
[image.shape[1]/2 - dst_size, image.shape[0] - 2*dst_size - bottom_offset],
])
# 2) Apply perspective transform
warped,mask = perspect_transform(Rover.img,source,destination)
# 3) Apply color threshold to identify navigable terrain/obstacles/rock samples
threshed = color_thresh(warped)
obs_map = np.absolute(np.float32(threshed) - 1 ) * mask
# 4) Update Rover.vision_image (this will be displayed on left side of screen)
# Example: Rover.vision_image[:,:,0] = obstacle color-thresholded binary image
# Rover.vision_image[:,:,1] = rock_sample color-thresholded binary image
# Rover.vision_image[:,:,2] = navigable terrain color-thresholded binary image
Rover.vision_image[:,:,2] = threshed * 255
Rover.vision_image[:,:,0] = obs_map * 255
# 5) Convert map image pixel values to rover-centric coords
xpix, ypix = rover_coords(threshed)
# 6) Convert rover-centric pixel values to world coordinates
world_size = Rover.worldmap.shape[0]
scale = 2 * dst_size
x_world, y_world = pix_to_world(xpix, ypix, Rover.pos[0], Rover.pos[1],Rover.yaw, world_size, scale)
obsxpix, obsypix = rover_coords(obs_map)
obs_x_world, obs_y_world = pix_to_world(obsxpix, obsypix, Rover.pos[0], Rover.pos[1],Rover.yaw, world_size, scale)
# 7) Update Rover worldmap (to be displayed on right side of screen)
# Example: Rover.worldmap[obstacle_y_world, obstacle_x_world, 0] += 1
# Rover.worldmap[rock_y_world, rock_x_world, 1] += 1
# Rover.worldmap[navigable_y_world, navigable_x_world, 2] += 1
Rover.worldmap[y_world, x_world, 2] += 10
Rover.worldmap[obs_y_world, obs_x_world, 0] += 1
# 8) Convert rover-centric pixel positions to polar coordinates
dist, angles = to_polar_coords(xpix,ypix)
# Update Rover pixel distances and angles
# Rover.nav_dists = rover_centric_pixel_distances
# Rover.nav_angles = rover_centric_angles
Rover.nav_angles = angles
rock_map = find_rocks(warped,levels=(110,110,50))
if rock_map.any():
rock_x, rock_y = rover_coords(rock_map)
rock_x_world, rock_y_world = pix_to_world(rock_x, rock_y, Rover.pos[0],
Rover.pos[1],Rover.yaw, world_size, scale)
rock_dist, rock_ang = to_polar_coords(rock_x, rock_y)
rock_idx = np.argmin(rock_dist)
rock_xcen = rock_x_world[rock_idx]
rock_ycen = rock_y_world[rock_idx]
Rover.worldmap[rock_ycen, rock_xcen, 1] = 255
Rover.vision_image[:,:,1] = rock_map * 255
Rover.vision_image[:,:,1] = 0
else:
Rover.vision_image[:,:,1] = 0
return Rover2. Launching in autonomous mode your rover can navigate and map autonomously. Explain your results and how you might improve them in your writeup.
I was able to launch the simulator and able to map 40% of environment and with more than 60% fidelity.
Note: running the simulator with different choices of resolution and graphics quality may produce different results, particularly on different machines! Make a note of your simulator settings (resolution and graphics quality set on launch) and frames per second (FPS output to terminal by drive_rover.py) in your writeup when you submit the project so your reviewer can reproduce your results.
Here I'll talk about the approach I took, what techniques I used, what worked and why, where the pipeline might fail and how I might improve it if I were going to pursue this project further.
