Christmas and Exam Break

This is the Final Update for a while, with the Christmas break and upcoming January exams it is difficult to devote time to the project

Work will Re-commence when the new term starts in February

Chris

Setting up the pockets

So far, the physics engine depends on the user clicking and dragging the mouse to control the speed and direction, this means that it can still be used for testing while the program is being developed.  


The pockets are set up using the getWidth() and getHeight() functions and the size of the radius.  The corner pockets are 2 radii wide by 2 radii high and the centre pockets are 4 radii wide.


Below is the code for the top left pocket

if((balls[i].position.getX() <= 2*balls[i].getRadius()) 
   && (balls[i].position.getY() <= 2*balls[i].getRadius()))


However we don't want the cue ball entering the pocket and being counted as a 'hit' so I added:

if((i>0) && (balls[i].position.getX() <= 2*balls[i].getRadius()) 
&& (balls[i].position.getY() <= 2*balls[i].getRadius()))
where the ball i=0 is the cue ball.

When this 'if' statement is fulfilled, the ball velocities are set to zero, a 1 is entered in the appropriate pocket of the hit array (in this case the hit array would be [1, 0, 0, 0, 0, 0] to indicate a hit in the first pocket), and the value of 'true' is returned to a later part of the code.

Interim Report

For the next few weeks focus will be on the Presentation and the Interim Report which is due in a few weeks time (14th Dec)

Chris

Next stages for the Physics Engine

The program needs to be evolved so it can:

• Set up the table with the balls already on it
• Detect if a ball enters a pocket and declare it a 'hit'
• Use the position and angle of the cue
• Maybe incorporate the strength of the hit
• Maybe calculate the best shot and rank possible 'hit's
• Pass information to the Server

PHP accepting FIle

The code which is responsible for sending the file to the server sends the file to a PHP file

Apache Tomcat doesn't run PHP

Within the MAC OS X operating system, there is the Apache HTTP sevrer which does run PHP but not by default, it must be turned on within the operating System

The Code was tested and worked for sending the file from the Phone to the server but the file is not displayed on the server

More work needs to be done on the server side script for the file.

A screenshot of the file upload working is shown below:

Progress on GUI

Due to Other Exam and Coursework Commitments progress on the GUI has been slowed, but this week is back on full speed.

I have been previously on the idea of a JSP form to upload an image manually, this has been created and has been taken one step further because now the uploaded image is passed into a prototype of the image analysis algorithm.

I have also found some open source code on how to send a file automatically from an android app, after fiddling with the code, it now sends a file to the server, but, so far, i havent been able to get the server side script to work which means the file isnt accepted, this is where i am spending most of my time in trying to get the image to send. I feel that this is an important part of the project.

Chris

Integrating File Upload with the Image Processing

The idea of the file upload was that it could then be passed to the Image Processing Algorithm

The first idea was to change the filename in the algorithm to match the uploaded file but this proved too complicated.

So, everyfile which was uploaded to the server was named the same and it would be overwritten everytime a file was sent to the server.

The file was sent to the algorithm but no new files created, the permissions need looking into

Chris

Ball Detection

There were 2 possible ways of detecting the balls:

1. Using the Hough Transform and 
2. By searching for white pixels in own algorithm

The Hough method didn't work so option 2 is being used.  With correct filtering, the canny image will leave a circle for each ball.  As it is a circle, the radius should be constant..


A vertical search is run for a white pixel then it continues searching for the next white pixel, it calculates the distance between them and this gives the diameter.  


The search then moves to the centre of these two white pixels and searches left and right.  If these measurements are the same as the vertical diameter, then the centre point is established.

Group Presentation

The group is set to make our presentation on 30th of November at 10:55 so now we are all concentrating on preparing our presentation and figuring out what we are going to say.

Grace

Hough Transform

This is used for finding straight lines in an image.  Instead of using Cartesian coordinates it uses polar coordinates to find straight lines on the image.


The Hough Transform can only be executed after the edge detector has been implemented.
Code by Edinburgh university was adapted for the relevant implementation.
http://homepages.inf.ed.ac.uk/rbf/HIPR2/hough.htm

http://homepages.inf.ed.ac.uk/rbf/HIPR2/flatjavasrc/Hough.java


In the first tests, the Hough lines were drawn incorrectly.  This could be due to the Hough matrix being wrong or the drawing of the Hough lines is wrong.

Uploading File to Server

Code was found at http://reecon.wordpress.com/2010/04/25/uploading-files-to-http-server-using-post-android-sdk/

This code sends a file to a server using methods within the android sdk.

This could be integrated into the application so that when a photo was taken, the file was sent to the server.

Chris

Running of the application

The application takes an image after 5 seconds and stores it in the correct location at a file size idea for the server.


Problem
When the timer is started and the XML screen is 'quitted' either by going back to the menu or leaving the app, the app crashes after 5 seconds as the thread for the timer is not correctly stopped.


Done:

• Background removal
• Edge detection
• Simple application to take an image after a timer

Starting to develop the Physics Engine

The first stage in developing the Physics Engine from the collisions code found during the research was to remove any extrac functions such as the control panel, the ball scattering and the fact that the colour changes with the velocity.

File Upload Problem

After research and work it was found that a File form cannot be autopopulated from a dynamic URL

This is due to security reasons in the JSP development. If it were allowed files could be taken from the users computer by the user just pressing a link.

Form has been returned to origanal state where the file is manually uploaded

Chris

Further research for the Physics Engine

Looking to expand on the previous model, through further research I found a different program in an article on Stack Overflow:

http://stackoverflow.com/questions/345838/ball-to-ball-collision-detection-and-handling

This uses round objects that collide with each other and the walls which can be adapted to suit the project specifications but needs a lot of work.

Passing a Variable to a Form

The Form below is being populated using the dynamic URL, the form box has the form string

Using the Camera

The camera application already contained on the phone cannot have a timer to set automatically and then take an image.


A possible solution is to use a surface view with a camera preview.  Open source code can be found by the blogger Android-er which implements this:
http://android-er.blogspot.com/2010/12/add-overlay-on-camera-preview.html


This method does not affect the predefined camera settings and can easily be adapted to incorporate a timer and specific resolution settings.

Physics Engine Research

The blog http://beginwithjava.blogspot.co.uk/ has several simple collision tutorials that introduce java methods.  

This is a good tutorial:

http://beginwithjava.blogspot.co.uk/2010/08/more-collision-detection-bouncing-right.html

and should help in the development of the physics engine.  It may be a good idea to produce a graphical representation of the pool game in order to ensure that it is behaving as predicted

This program contains a moving object (the green square) colliding with the walls of the window as well as the green square. 

Physics Engine Prototype Review

It was only designed to be a basic program but it allowed straight collisions and didn't account for the walls of the pool table or whether the pockets were in a realistic position.

However, the program was able to receive values for the coordinates of the balls, cue, and pockets from the GUI, perform the calculations, and then inform the GUI section whether the shot will be a hit or a miss.

Edge Detection Implementation

When I was looking for examples of canny edge detection implementations in Java, I found code developed by Tom Gibara:

http://www.tomgibara.com/computer-vision/canny-edge-detector

This implementation takes an image RGB, grayscale and performs a canny edge detection and returns a buffered image.

The useful code to perform the canny was extracted and implemented with the image produced from the background removal.  The output can then be compared with the result obtained from Matlab.

Canny Edge Detector

A Gaussian filter is used to reduce the noise.  The image is convolved with the Gaussian filter


Next is to find the edge and determine the direction and gradient.  Four Sobel masks are used to find the edge gradient and the sum of all four is found and then the magnitude and angle can be found.


Using threshold value for the gradient and magnitude, the edges are filtered out


Then trace along the lines to give the canny image.

The code used in this section was created by Tom Gibara http://www.tomgibara.com/computer-vision/canny-edge-detector

Dynamic URL's

It is desired that the file is automatically uploaded to the server. One concept is to use a dynamic URL which would automatically populate the form on the page.

Dynamic URL's have the Form:

http://www.localhost:8080?value=object

value is the variable
object is its data value

The variable could then be extracted to the JSP using the code:

<% String Name = request.getParameter("value")%>

Initial image analysis

Once the image has been taken using the app, it is converted into a grayscale image using Matlab.  Peaks  with an intensity above 4000 between 54, 85 and 92, 149 where these values correspond to the grey colour value.  Removing these colour shows 2 out of the 5 balls.

Using each colour intensity RGB components, histograms for each pixel were made.  All of the green components were removed.  Using a maximum intensity of 25000, the red values cut off 67-133 and 67-135 gave better values with all 5 balls found but white patches on the table.

Changing the red and blue bands to give the best results, 5 balls found and edge of the table, the felt was blocked out.  If it finds a red between 85 and 151, a blue between 47 and 170, and green between 0 and 255, these values are set to zero and all other values set to 255 (white)

Matlab has a function 'edge' which can apply different edge detection techniques: canny, sobel, log, roberts, prewitt, and zero cross.

The best results were found using the canny technique.

Geometry Section - Physics Engine

The links on this page should be very useful for the pool engine

http://gamedev.stackexchange.com/questions/7862/is-there-an-algorithm-for-a-pool-game

File Upload Form

A form was made on the server so that the user could manually upload their photo to the server

Making a prototype

In order to ensure that we can make a basic program to complete the task, we decided that it is important to design and make a prototype.

For the Physics Engine section, a very simple program to calculate if the balls and cue are directly lined up with a pocket, using gradient calculations.  Another challenge will be to get the three distinct sections to transfer information between them.

The deadline for this prototype is next Wednesday (2nd November).

Grace

Server Requirements

The server must be able to do the following tasks:

• Accept Image from the Mobile Device
• Pass Image to Image Analysis which would receive Co-ordinates and Angles
• Process approx 30 variables
• Pass Values to the engine
• Take Output values and display a hit or miss to user

Apache Tomcat Web Server was chosen, with pages running JSP's  http://tomcat.apache.org/

The image analysis and physics engine were to be written in Java

Stages in the Data Acquisition and Image Analysis

1. Unwarp the image
2. Background removal
3. Ball detection
4. Table Outline detection
• rotation/ grid/ coordinates

Possible Server Languages

Client Side Languages->Program is performed by the web browser using the users processing


Server Side->Program is Run on the server and output is displayed to the user, this is the preferred option

Server Side languages include:

• Java via JSP pages
• Javascript
• PHP
• Python

Using Matlab

Found Matlab function for colour detection.  Image file is opened and converted to RGB values.  These values are analysed and if they are between a given range, a 2D array for the given pixel is given an integer value.  Depending on the integer value, the dimensions of the ball can be calculated.

Another example Matlab function was found called Edge Detection.  This function finds the edges of coins and puts a green line around each coin.

Camera Preview and Save Image Blogs

 http://android-er.blogspot.com/2010/12/add-overlay-on-camera-preview.html

http://android-er.blogspot.com/2010/12/implement-takepicture-function-of.html

http://android-er.blogspot.com/2011/01/save-camera-image-using-mediastore.html

These three links by android-er blog are very useful for taking an image and saving the image to the sd card using a surfaceHolder (image preview) without using the camera application

Physics Engine Requirements

The Physics Engine must complete the following tasks:

• Receive the information passed from the image capture.
• Calculate whether or not the ball which is being aimed at is going to be hit.
• Calculate a potential shot in which the ball will be potted.
• Calculate amount of correction needed in the angle of the cue to achieve potential shot.
• Pass information to user interface

Ideally, the first task should be to find an existing model that can be adapted to fit the task requirements.

Android Development Tutorials

  http://www.youtube.com/user/thenewboston#grid/user/2F07DBCDCC01493A

This is a great page with 200 tutorials on developing for Android.
We've found this very useful, and really explains it for complete Beginners like us

Considerations, Problems and Algorithm

Graphical User Interface: Problems and Deliverables

Deliverables:

The GUI needs to be functional in its operation in linking the user to the algorithms and calculations contained within the application. This needs to be simple and intuitive to use for a new user and appealing and easy pleasing to a regular user.

The GUI also acts as the controller for the other functions (Image Capture and Geometry) and calls them when the user requires them and uses info from them to tell the user if the shot is going to be a hit or miss.

 

The GUI will also take the information from the geometry, and passes it on to the user in the form of a ‘beep’. This ‘beep’ will be calibrated so that the gaps get smaller the closer the shot is to the pocket, until a constant ring when the shot will be a hit.

Expected Problems:

•  Application runs too slowly for it to be useable, time will need to be spent improving the speed and reduce the size of the program.

•  Bugs in the code prevent it from running; time will need to be spent testing program and all possibilities within it to weed out problems.

 Potential Problems:

•  Information passed to Image capture is wrong or not detailed enough.

•  Information from Geometry is wrong or incorrect.

•  Design is not functional or eye catching

•  Completion of code does not meet time schedule

•  The GUI is functional, but when other sections are added it no longer is.

 

 Information to be passed to Image Capture:

•  Turning on the camera and preparing it for image capture

Information to be received from Geometry:

•  Whether the shot is going to be a hit or miss

•  If the shot is a miss, the distance the shot is away from the pocket

 

Steps in the Design and Implementation of the GUI:

•  Two Algorithms designed, one which starts with user and ends with image capture and another which starts with geometry and ends with ‘beep’ to user.

•  A design drawn out and agreed upon

•  Design created and coded, as an empty shell with no functions within

•  Design approved

•  Main function added i.e. Screen shows the image seen by the camera and the image capture function is called upon.

•  Noise alert function added

•  Noise alert function tested using self inputted numbers

•  Noise alert implemented with numbers from Geometry function

•  Testing application with full implementation of the image capture and geometry.

Optional Extras:

•  Options menu for user to personalize the way the application is run or shown

•  Recorded statistics for the users hit, miss ratio etc

•  Use of accelerometer within the phone to make sure the phone is flat

•  Use of web server to display the image seen by the camera on a computer screen, and line of the expected path plotted 

Algorithm for the GUI

  Image Capture and analysis: Problems and Deliverables

Deliverables:

The image capture and analysis needs to efficiently obtain an image using the mobile phones camera when held above the pool table. Once this image has been captured the analysis can be undertaken. From the image analysis the table’s dimensions such as the edges and pockets need to be defined. Also each ball needs to be located on the table and each ball differentiated from each other, the angle and direction of the cue need to be defined. All this information must be sent to the geometry calculation section so the shot assistant can be calculated.

Problems

•    Not enough of the table is ‘in view’ using the camera so the shot cannot be predicted when the data is passed to the geometry calculation.

•  Data from the image analysis could be in the wrong format for the geometry calculations

•   None of the balls could be detected.

•  The mobile phone is not level so that the image analysis creates incorrect locations for the balls and cue.

•    Mobile phone is moved about to much so that an accurate image could not be taken

·  

Information to be sent to Geometry Calculations:

The geometry calculation sections will need the following information so that the calculations for the shot can be executed. They are;

•  The table’s dimensions such as the boundary of the table and the location of each of the pockets.

•  The different ball types such as the difference between the coloured and striped balls and the cue ball which is white.

•  Ball location, the position at which each ball is on the table.

•  The initial cue angle held by the player.

Information to be sent to the GUI:

•  The new image needs to be displayed on the screen of the mobile phone.

•  If the image being captured is not big enough for the full analysis I.e. the edges of the table of pockets could not be seen.

Information to be received from GUI:

•  The GUI will prompt the image capture section when the camera needs to be turned on.

•  Image capture algorithm will be prompted to be executed by the GUI.  

•  The GUI will send the sounds. 

Steps in the Design and Implementation of the Image capture and analysis:

•  Create a program which takes an image with the mobile phone’s camera ready for the analysis.

•  Create and implement an algorithm for defining each the table dimensions.

•  Choose an appropriate way of storing the information which will be sent for the geometry calculations.

•  Create and implement an algorithm to distinguish the different types of balls and the balls location on the table.

•  Create and implement and algorithm to detect the angle and direction of the cue in relation to a specific ball. 

•  Send actual information the geometry calculations section

Algorithm for the Image Capture

Geometry Calculations: Problems and Deliverables

Deliverables:

Geometry Calculations section is required to take the information passed from the image capture and analysis and calculates whether or not the ball which is being aimed at is going to be hit. Once the ball has been targeted an algorithm needs to calculate a potential shot in which the ball will be potted and the amount of correction needed by changing the angle of the cue in relation to the cue ball.  

Expected Problems:

•   Balls cannot be lined up with hole (no direct shot).

•  Confused by 2 balls on table – aiming for wrong one but correct colour.

•   Aiming for wrong user’s ball.

Potential Problems:

•   Code runs too slowly to calculate correct cue position in the allocated time.

•   Program calculates balls heading in wrong direction.

Information to be sent to the GUI:

•    Whether the current shot is going to be a hit or a miss.

•   The distance the cue needs to be moved to correct the shot.

Information to be received from image capture and analysis:

•   The dimensions of the pool table such as the edges and the location of each pocket.

•   The location of each ball and what type of ball they are.

•   The angle of the cue with relation the cue ball.

Steps in the Design and Implementation of the geometry calculation:

 •  Create calculations from collision theory to predict which direction the balls will move in.
 •  Design an algorithm with for the collision movement.
 •  Create the program and test with ‘fake’ values.
 •  Test with real values obtained from the image capture and analysis. 
 •  Send the proposed improvements to the GUI.

Algorithm for the Geometry

Data Acquisition and Image Analysis Requirements

The following tasks are what is required:

• Automated Data Acquisition
• Automatic Detection of ball the positions
• Automatic Detection of the table
• Automatic Detection of the cue and its angle to the ball

Pool robot project at Queens

Learning the Basics

In order to learn the skills we will need for this project, we have been following tutorials on Youtube about creating android applications.


Within the project we will be using Eclipse and Android 2.2


We will learn about:

• XML layouts
• Threads
• Using different screens
• Using the camera
• Buttons
• Timers

Andy